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Preface 


INTENDED AUDIENCE 

This book originally grew out of our lecture notes for an “Applied Multivariate 
Analysis” course offered jointly by the Statistics Department and the School of 
Business at the University of Wisconsin-MadisonApp/ierf Multivariate Statisti¬ 
cal Analysis, Sixth Edition, is concerned with statistical methods for describing and 
analyzing multivariate data. Data analysis, while interesting with one variable, 
becomes truly fascinating and challenging when several variables are involved. 
Researchers in the biological, physical, and social sciences frequently collect mea¬ 
surements on several variables. Modern computer packages readily provide the 
numerical results to rather complex statistical analyses. We have tried to provide 
readers with the supporting knowledge necessary for making proper interpreta¬ 
tions, selecting appropriate techniques, and understanding their strengths and 
weaknesses. We hope our discussions will meet the needs of experimental scien¬ 
tists, in a wide variety of subject matter areas, as a readable introduction to the 
statistical analysis of multivariate observations. 


LEVEL 


Our aim is to present the concepts and methods of multivariate analysis at a level 
that is readily understandable by readers who have taken two or more statistics 
courses. We emphasize the applications of multivariate methods and, conse¬ 
quently, have attempted to make the mathematics as palatable as possible. We 
avoid the use of calculus. On the other hand, the concepts of a matrix and of ma¬ 
trix manipulations are important. We do not assume the reader is familiar with 
matrix algebra. Rather, we introduce matrices as they appear naturally in our 
discussions, and we then show how they simplify the presentation of multivari¬ 
ate models and techniques. 

The introductory account of matrix algebra, in Chapter 2, highlights the 
more important matrix algebra results as they apply to multivariate analysis. The 
Chapter 2 supplement provides a summary of matrix algebra results for those 
with little or no previous exposure to the subject. This supplementary material 
helps make the book self-contained and is used to complete proofs. The proofs 
may be ignored on the first reading. In this way we hope to make the book ac¬ 
cessible to a wide audience. 

In our attempt to make the study of multivariate analysis appealing to a 
large audience of both practitioners and theoreticians, we have had to sacrifice 
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a consistency of level. Some sections are harder than others. In particular, we 
have summarized a voluminous amount of material on regression in Chapter 7. 
The resulting presentation is rather succinct and difficult the first time through. 
We hope instructors will be able to compensate for the unevenness in level by ju¬ 
diciously choosing those sections, and subsections, appropriate for their students 
and by toning them clown if necessary. 


ORGANIZATION AND APPROACH 

The methodological “tools’" of multivariate analysis are contained in Chapters 5 
through 12 . These chapters represent the heart of the book, but they cannot be 
assimilated without much of the material in the introductory Chapters 1 through 
4 Even those readers with a good knowledge of matrix algebra or those willing 
to accept the mathematical results on faith should, at the very least, peruse Chap¬ 
ter 3 “Sample Geometry,” and Chapter 4, “Multivariate Normal Distribution.” 

’ Q ur approach in the methodological chapters is to keep the discussion di¬ 
rect and uncluttered. Typically, we start with a formulation of the population 
models, delineate the corresponding sample results, and liberally illustrate every¬ 
thing with examples. The examples are of two types: those that are simple and 
whose calculations can be easily done by hand, and those that rely on real-world 
data and computer software. These will provide an opportunity to (1) duplicate 
our analyses, (2) carry out the analyses dictated by exercises, or (3) analyze the 
data using methods other than the ones we have used or suggested. 

The division of the methodological chapters (5 through 12) into three units 
allows instructors some flexibility in tailoring a course to their needs. Possible 
sequences for a one-semester (two quarter) course are indicated schematically. 

Each instructor will undoubtedly omit certain sections from some chapters 
to cover a broader collection of topics than is indicated by these two choices. 



For most students, we would suggest a quick pass through the first four 
chapters (concentrating primarily on the material in Chapter 1; Sections 2.1,2.2, 
2 3 2 5,2.6, and 3.6; and the “assessing normality” material in Chapter 4) fol¬ 
lowed by a'selection of methodological topics. For example, one might discuss 
the comparison of mean vectors, principal components, factor analysis, discrimi¬ 
nant analysis and clustering. The discussions could feature the many “worked 
out” examples included in these sections of the text. Instructors may rely on di- 
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agrams and verbal descriptions to teach the corresponding theoretical develop¬ 
ments. If the students have uniformly strong mathematical backgrounds, much of 
the book can successfully be covered in one term. 

We have found individual data-analysis projects useful for integrating ma¬ 
terial from several of the methods chapters. Here, our rather complete treatments 
of multivariate analysis of variance (MANOVA), regression analysis, factor analy¬ 
sis, canonical correlation, discriminant analysis, and so forth are helpful, even 
though they may not be specifically covered in lectures. 


CHANGES TO THE SIXTH EDITION 



New material. Users of the previous editions will notice several major changes 
in the sixth edition. 

• Twelve new data sets including national track records for men and women, 
psychological profile scores, car body assembly measurements, cell phone 
tower breakdowns, pulp and paper properties measurements, Mali family 
farm data, stock price rates of return, and Concho water snake data. 

• Thirty seven new exercises and twenty revised exercises with many of these 
exercises based on the new data sets. 

• Four new data based examples and fifteen revised examples. 

• Six new or expanded sections: 

1. Section 6.6 Testing for Equality of Covariance Matrices 

2. Section 11.7 Logistic Regression and Classification 

3. Section 12.5 Clustering Based on Statistical Models 

4. Expanded Section 6.3 to include “An Approximation to the Distrib¬ 
ution of T 2 for Normal Populations When Sample Sizes are not Large” 

5. Expanded Sections 7.6 and 7.7 to include Akaike’s Information Cri¬ 
terion 

6 . Consolidated previous Sections 11.3 and 11.5 on two group discrimi¬ 
nant analysis into single Section 11.3 

Web Site. To make the methods of multivariate analysis more prominent 
in the text, we have removed the long proofs of Results 7.2,7.4,7.10 and 10.1 
and placed them on a web site accessible through www.prenhall.com/statistics . 
Click on “Multivariate Statistics” and then click on our book. In addition, all 
full data sets saved as ASCII files that are used in the book are available on 
the web site. 

Instructors' Solutions Manual. An Instructors Solutions Manual is available 
on the author’s website accessible through www.prenhall.com/statistics. For infor¬ 
mation on additional for-sale supplements that may be used with the book or 
additional titles of interest, please visit the Prentice Hall web site at www.pren- 
halicom. 
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Chapter 



Aspects of Multivariate 
Analysis 

l.l Introduction 

Scientific inquiry is an iterative learning process. Objectives pertaining to the expla¬ 
nation of a social or physical phenomenon must be specified and then tested by 
gathering and analyzing data. In turn, an analysis of the data gathered by experi¬ 
mentation or observation will usually suggest a modified explanation of the phe¬ 
nomenon. Throughout this iterative learning process, variables are often added or 
deleted from the study. Thus, the complexities of most phenomena require an inves¬ 
tigator to collect observations on many different variables. This book is concerned 
with statistical methods designed to elicit information from these kinds of data sets. 
Because the data include simultaneous measurements on many variables, this body 
of methodology is called multivariate analysis. 

The need to understand the relationships between many variables makes multi¬ 
variate analysis an inherently difficult subject. Often, the human mind is over¬ 
whelmed by the sheer bulk of the data. Additionally, more mathematics is required 
to derive multivariate statistical techniques for making inferences than in a univari¬ 
ate setting. We have chosen to provide explanations based upon algebraic concepts 
and to avoid the derivations of statistical results that require the calculus of many 
variables. Our objective is to introduce several useful multivariate techniques in a 
clear manner, making heavy use of illustrative examples and a minimum of mathe¬ 
matics. Nonetheless, some mathematical sophistication and a desire to think quanti¬ 
tatively will be required. 

Most of our emphasis will be on the analysis of measurements obtained with¬ 
out actively controlling or manipulating any of the variables on which the mea¬ 
surements are made. Only in Chapters 6 and 7 shall we treat a few experimental 
plans (designs) for generating data that prescribe the active manipulation of im¬ 
portant variables. Although the experimental design is ordinarily the most impor¬ 
tant part of a scientific investigation, it is frequently impossible to control the 
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generation of appropriate data in certain disciplines. (This is true, for example, in 
business, economics, ecology, geology, and sociology.) You should consult [6] and 
[7] for detailed accounts of design principles that, fortunately, also apply to multi¬ 
variate situations. 

It will become increasingly clear that many multivariate methods are based 
upon an underlying probability model known as the multivariate normal distribution. 
Other methods are ad hoc in nature and are justified by logical or commonsense 
arguments. Regardless of their origin, multivariate techniques must, invariably, 
be implemented on a computer. Recent advances in computer technology have 
been accompanied by the development of rather sophisticated statistical software 
packages, making the implementation step easier. 

Multivariate analysis is a “mixed bag.” It is difficult to establish a classification 
scheme for multivariate techniques that is both widely accepted and indicates the 
appropriateness of the techniques. One classification distinguishes techniques de¬ 
signed to study interdependent relationships from those designed to study depen¬ 
dent relationships. Another classifies techniques according to the number of 
populations and the number of sets of variables being studied. Chapters in this text 
are divided into sections according to inference about treatment means, inference 
about covariance structure, and techniques for sorting or grouping. This should not, 
however, be considered an attempt to place each method into a slot. Rather, the 
choice of methods and the types of analyses employed are largely determined by 
the objectives of the investigation. In Section 1.2, we list a smaller number of 
practical problems designed to illustrate the connection between the choice of a sta¬ 
tistical method and the objectives of the study. These problems, plus the examples in 
the text, should provide you with an appreciation of the applicability of multivariate 
techniques across different fields. 

The objectives of scientific investigations to which multivariate methods most 
naturally lend themselves include the following: 

L Data reduction or structural simplification . The phenomenon being studied is 
represented as simply as possible without sacrificing valuable information. It is 
hoped that this will make interpretation easier. 

2. Sorting and grouping . Groups of “similar” objects or variables are created, 
based upon measured characteristics. Alternatively, rules for classifying objects 
into well-defined groups may be required. 

3. Investigation of the dependence among variables. The nature of the relation¬ 
ships among variables is of interest. Are all the variables mutually independent 
or are one or more variables dependent on the others? If so, how? 

4. Prediction. Relationships between variables must be determined for the pur¬ 
pose of predicting the values of one or more variables on the basis of observa¬ 
tions on the other variables. 

5. Hypothesis construction and testing . Specific statistical hypotheses, formulated 
in terms of the parameters of multivariate populations, are tested. This may be 
done to validate assumptions or to reinforce prior convictions. 

We conclude this brief overview of multivariate analysis with a quotation from 
F. H. C. Marriott [19], page 89. The statement was made in a discussion of cluster 
analysis, but we feel it is appropriate for a broader range of methods. You should 
keep it in mind whenever you attempt or read about a data analysis. It allows one to 
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maintain a proper perspective and not be overwhelmed by the elegance of some of 
the theory: 

If the results disagree with informed opinion, do not admit a simple logical interpreta¬ 
tion, and do not show up clearly in a graphical presentation, they are probably wrong. 
There is no magic about numerical methods, and many ways in which they can break 
down. They are a valuable aid to the interpretation of data, not sausage machines 
automatically transforming bodies of numbers into packets of scientific fact. 


1.2 Applications of Multivariate Techniques 

The published applications of multivariate methods have increased tremendously in 
recent years. It is now difficult to cover the variety of real-world applications of 
these methods with brief discussions, as we did in earlier editions of this book. How¬ 
ever, in order to give some indication of the usefulness of multivariate techniques, 
we offer the following short descriptions.of the results of studies from several disci¬ 
plines. These descriptions are organized according to the categories of objectives 
given in the previous section. Of course, many of our examples are multifaceted and 
could be placed in more than one category. 

Data reduction or simplification 

• Using data on several variables related to cancer patient responses to radio¬ 
therapy, a simple measure of patient response to radiotherapy was constructed. 
(See Exercise 1.15.) 

• Track records from many nations were used to develop an index of perfor¬ 
mance for both male and female athletes. (See [8] and [22].) 

• Multispectral image data collected by a high-altitude scanner were reduced to a 
form that could be viewed as images (pictures) of a shoreline in two dimensions. 
(See [23].) 

• Data on several variables relating to yield and protein content were used to cre¬ 
ate an index to select parents of subsequent generations of improved bean 
plants (See [13].) 

• A matrix of tactic similarities was developed from aggregate data derived from 
professional mediators From this matrix the number of dimensions by which 
professional mediators judge the tactics they use in resolving disputes was 
determined. (See [21].) 

Sorting and grouping 

• Data on several variables related to computer use were employed to create 
clusters of categories of computer jobs that allow a better determination of 
existing (or planned) computer utilization. (See [2].) 

• Measurements of several physiological variables were used to develop a screen¬ 
ing procedure that discriminates alcoholics from nonalcoholics (See [26].) 

• Data related to responses to visual stimuli were used to develop a rule for sepa¬ 
rating people suffering from a multiple-sclerosis-caused visual pathology from 
those not suffering from the disease. (See Exercise 1.14.) 
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• The U.S. Internal Revenue Service uses data collected from tax returns to sort 
taxpayers into two groups: those that will be audited and those that will not. 
(See [31].) 

Investigation of the dependence among variables 

• Data on several variables were used to identify factors that were responsible for 
client success in hiring external consultants. (See [12].) 

• Measurements of variables related to innovation, on the one hand, and vari¬ 
ables related to the business environment and business organization, on the 
other hand, were used to discover why some firms are product innovators and 
some firms are not. (See [3].) 

• Measurements of pulp fiber characteristics and subsequent measurements of 
characteristics of the paper made from them are used to examine the relations 
between pulp fiber properties and the resulting paper properties. The goal is to 
determine those fibers that lead to higher quality paper. (See [17].) 

• The associations between measures of risk-taking propensity and measures of 
socioeconomic characteristics for top-level business executives were used to 
assess the relation between risk-taking behavior and performance. (See [18].) 

Prediction 

• The associations between test scores, and several high school performance vari¬ 
ables, and several college performance variables were used to develop predic¬ 
tors of success in college. (See [10].) 

• Data on several variables related to the size distribution of sediments were used to 
develop rules for predicting different depositional environments. (See [7] and [20].) 

• Measurements on several accounting and financial variables were used to de¬ 
velop a method for identifying potentially insolvent property-liability insurers. 
(See [28].) 

• cDNA microarray experiments (gene expression data) are increasingly used to 
study the molecular variations among cancer tumors. A reliable classification of 
tumors is essential for successful diagnosis and treatment of cancer. (See [9].) 

Hypotheses testing 

• Several pollution-related variables were measured to determine whether levels 
for a large metropolitan area were roughly constant throughout the week, or 
whether there was a noticeable difference between weekdays and weekends. 
(See Exercise 1.6.) 

• Experimental data on several variables were used to see whether the nature of 
the instructions makes any difference in perceived risks, as quantified by test 
scores. (See [27].) 

• Data on many variables were used to investigate the differences in structure of 
American occupations to determine the support for one of two competing soci¬ 
ological theories. (See [16] and [25].) 

• Data on several variables were used to determine whether different types of 
firms in newly industrialized countries exhibited different patterns of innova¬ 
tion. (See [15].) 
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The preceding descriptions offer glimpses into the use of multivariate methods 
in widely diverse fields. 


1.3 The Organization of Data 

Throughout this text, we are going to be concerned with analyzing measurements 
made on several variables or characteristics. These measurements (commonly called 
data) must frequently be arranged and displayed in various ways. For example, 
graphs and tabular arrangements are important aids in data analysis. Summary num¬ 
bers, which quantitatively portray certain features of the data, are also necessary to 
any description. 

We now introduce the preliminary concepts underlying these first steps of data 
organization. 

Arrays 

Multivariate data arise whenever an investigator, seeking to understand a social or 
physical phenomenon, selects a number p > 1 of variables or characters to record. 
The values of these variables are all recorded for each distinct item , individual , or 
experimental unit . 

We will use the notation x ]k to indicate the particular value of the /cth variable 
that is observed on the yth item, or trial. That is, 

x jk = measurement of the fcth variable on the yth item 

Consequently, n measurements on p variables can be displayed as follows: 



Variable 1 

Variable 2 

Variable k 

Variable p 

Item 1: 

*n 

*12 

*i k 

*i P 

Item 2: 

*21 

*22 

*2 k 

*2 p 

Item j: 

x n 

X J2 

x jk 

x jp 

Item n: 

*ni 

x n2 

x nk 

x np 


Or we can display these data as a rectangular array, called X, of n rows and p 
columns: 

*11 *12 *** *1 k **■ *1 p 

*21 *22 *" *2 k *2 p 

x = 1 1 : : 

*yi *y2 * * * x jk * ’ • x j P 

_*n 1 x n2 *** x nk ’** x np 

The array X, then, contains the data consisting of all of the observations on all of 
the variables. 
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Example l.l (A data array) A selection of four receipts from a university bookstore 
was obtained in order to investigate the nature of book sales. Each receipt provided, 
among other things, the number of books sold and the total amount of each sale. Let 
the first variable be total dollar sales and the second variable be number of books 
sold. Then we can regard the corresponding numbers on the receipts as four mea¬ 
surements on two variables. Suppose the data, in tabular form, are 

Variable 1 (dollar sales): 42 52 48 58 

Variable 2 (number of books): 4 5 4 3 

Using the notation just introduced, we have 

*i i = 42 X 21 ~ 52 JC31 = 48 JC41 = 58 

X 12 = 4 X 2 2 = 5 *32 = 4 *42 = 3 

and the data array X is 

"42 4~ 

Y _ 52 5 

48 4 
58 3 



& 


I 


with four rows and two columns. ■ 

Considering data in the form of arrays facilitates the exposition of the subject 
matter and allows numerical calculations to be performed in an orderly and efficient 
manner. The efficiency is twofold, as gains are attained in both (1) describing nu¬ 
merical calculations as operations on arrays and (2) the implementation of the cal¬ 
culations on computers, which now use many languages and statistical packages to 
perform array operations. We consider the manipulation of arrays of numbers in 
Chapter 2. At this point, we are concerned only with their value as devices for dis¬ 
playing data. 
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If the n measurements represent a subset of the full set of measurements that 
might have been observed, then * t is also called the sample mean for the first vari¬ 
able. We adopt this terminology because the bulk of this book is devoted to proce¬ 
dures designed to analyze samples of measurements from larger collections. 

The sample mean can be computed from the n measurements on each of the 
p variables, so that, in general, there will be p sample means: 

= k = l,2,...,p (1-1) 

" J =1 

A measure of spread is provided by the sample variance , defined for n measure¬ 
ments on the first variable as 


~ 2 (*;i - *i) 2 
n /=1 

where x\ is the sample mean of the * ;1 ’s. In general, for p variables, we have 

4 “ i 2 (*/* - x*) 2 k = 1,2,... ,p (1-2) 

n /=1 

Two comments are in order. First, many authors define the sample variance with a 
divisor of n — 1 rather than n. Later we shall see that there are theoretical reasons 
for doing this, and it is particularly appropriate if the number of measurements, n y is 
small. The two versions of the sample variance will always be differentiated by dis¬ 
playing the appropriate expression. 

Second, although the s 2 notation is traditionally used to indicate the sample 
variance, we shall eventually consider an array of quantities in which the sample vari¬ 
ances lie along the main diagonal. In this situation, it is convenient to use double 
subscripts on the variances in order to indicate their positions in the array. There¬ 
fore, we introduce the notation s kk to denote the same variance computed from 
measurements on the A:th variable, and we have the notational identities 


Descriptive Statistics 

A large data set is bulky, and its very mass poses a serious obstacle to any attempt to 
visually extract pertinent information. Much of the information contained in the 
data can be assessed by calculating certain summary numbers, known as descriptive 
statistics. For example, the arithmetic average, or sample mean, is a descriptive sta¬ 
tistic that provides a measure of location—that is, a “central value” for a set of num¬ 
bers. And the average of the squares of the distances of all of the numbers from the 
mean provides a measure of the spread, or variation, in the numbers. 

We shall rely most heavily on descriptive statistics that measure location, varia¬ 
tion, and linear association. The formal definitions of these quantities follow. 

Let *u, *21,.. ■, *„i be n measurements on the first variable. Then the arith¬ 
metic average of these measurements is 

1 n 

*1 = " 2 x n 

n y„i 


4 = *** “ £ 2 (*/* - X*) 2 ^ = 1,2,..., p (1-3) 

11 /= 1 

The square root of the sample variance, Vs^, is known as the sample standard 
deviation. This measure of variation uses the same units as the observations. 

Consider n pairs of measurements on each of variables 1 and 2: 



That is, Xji and * ;2 are observed on the ;'th experimental item (j = 1, 2,...,«). A 
measure of linear association between the measurements of variables 1 and 2 is pro¬ 
vided by the sample covariance 

1 n 

S 12 ~ “ 2 ( X J 1 ~ * l )(*/2 “ * 2 ) 

" /=! 
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or the average product of the deviations from their respective means. If large values for 
one variable are observed in conjunction with large values for the other variable, and 
the small values also occur together, s 12 will be positive. If large values from one vari¬ 
able occur with small values for the other variable, s 12 will be negative. If there is no 
particular association between the values for the two variables, s i2 will be approxi¬ 
mately zero. 

The sample covariance 

1 » 

s ik ~ (*/«■ ~ ■**) i X jk x k ) * 1, 2, ..., / 7 , k 1> 2, ... , p (1-4) 

measures the association between the ith and fcth variables.We note that the covari¬ 
ance reduces to the sample variance when 1 = k. Moreover, s ik = s ki for all i and k. 

The final descriptive statistic considered here is the sample correlation coeffi¬ 
cient (or Pearson's product-moment correlation coefficient, ; see [14]). This measure 
of the linear association between two variables does not depend on the units of 
measurement. The sample correlation coefficient for the ith and £th variables is 
defined as 


n 



for i = 1,2,...,/? and k — 1,2,...,/?. Note r ik ~ r ki for all i and k . 

The sample correlation coefficient is a standardized version of the sample co- 
variance, where the product of the square roots of the sample variances provides the 
standardization. Notice that r ik has the same value whether?: or n — 1 is chosen as 
the common divisor for s iiy s kk , and s ik . 

The sample correlation coefficient r ik can also be viewed as a sample covariance. 
Suppose the original values Jty, and x jk are replaced by standardized values 
( xji - Xi)/ Vsfi and (x jk - x k )/ Vs^ k . The standardized values are commensurable be¬ 
cause both sets are centered at zero and expressed in standard deviation units. The sam¬ 
ple correlation coefficient is just the sample covariance of the standardized observations. 

Although the signs of the sample correlation and the sample covariance are the 
same, the correlation is ordinarily easier to interpret because its magnitude is 
bounded. To summarize, the sample correlation r has the following properties: 

1. The value of rmust be between -1 and +1 inclusive. 

2. Here r measures the strength of the linear association. If r = 0, this implies a 
lack of linear association between the components. Otherwise, the sign of r indi¬ 
cates the direction of the association: r < 0 implies a tendency for one value in 
the pair to be larger than its average when the other is smaller than its average; 
and r > 0 implies a tendency for one value of the pair to be large when the 
other value is large and also for both values to be small together. 

3. The value of r ik remains unchanged if the measurements of the fth variable 
are changed to y ;7 = axj t + bj = 1,2,..., n, and the values of the kth vari¬ 
able are changed to y jk = cxj k + d,j = 1 ,provided that the con¬ 
stants a and c have the same sign. 
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The quantities s ik and r lk do not, in general, convey all there is to know about 
the association between two variables. Nonlinear associations can exist that are not 
revealed by these descriptive statistics. Covariance and correlation provide mea¬ 
sures of linear association, or association along a line. Their values are less informa¬ 
tive for other kinds of association. On the other hand, these quantities can be very 
sensi ive to wi observations ( outliers”) and may indicate association when, in 
tact, little exists. In spite of these shortcomings, covariance and correlation coeffi¬ 
cients are routinely calculated and analyzed. They provide cogent numerical sum¬ 
maries of association when the data do not exhibit obvious nonlinear patterns of 
association and when wild observations are not present. 

Suspect observations must be accounted for by correcting obvious recording 
mistakes and by taking actions consistent with the identified causes. The values of 
s ik and r ik should be quoted both with and without these observations. 

The sum of squares of the deviations from the mean and the sum of cross- 
product deviations are often of interest themselves. These quantities are 

n 

w kk = 2 ( x ik ~x k ) 2 k = 1, 2 ,.... p (1-6) 

and 

n 

W,k = 2 ( x ji ~ x i) ( x jk ~ *k) i = 1 , 2 ,.... p, k = 1,2,..., p ( 1 - 7 ) 

The descriptive statistics computed from n measurements on p variables can 
also be organized into arrays. 

Arrays of Basic Descriptive Statistics 


Sample means x = 
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The sample mean array is denoted by x, the sampJe variance and covariance 
array by the capital letter S n , and the sample correlation array by R.The subscript n 
on the array S„ is a mnemonic device used to remind you that n is employed as a di¬ 
visor for the elements s ik . The size of all of the arrays is determined by the number 
of variables, p. 

The arrays S„ and R consist of p rows and p columns. The array x is a single 
column with p rows. The first subscript on an entry in arrays S„ and R indicates 
the row; the second subscript indicates the column. Since s ik = s ki and r ik = r ki 
for all i and k, the entries in symmetric positions about the main northwest- 
southeast diagonals in arrays S„ and R are the same, and the arrays are said to be 
symmetric . 

Example 1.2 (The arrays x, S n> and R for bivariate data) Consider the data intro¬ 
duced in Example 1.1. Each receipt yields a pair of measurements, total dollar 
sales, and number of books sold. Find the arrays x, S„, and R. 

Since there are four receipts, we have a total of four measurements (observa¬ 
tions) on each variable. 

The sample means are 

4 

i, = j X x ;i = 3( 42 + 52 + 48 + 58) = 50 
;=i 

x 2 = \ x x i2 = 3(4 + 5 + 4 + 3) = 4 
;=i 

Xi 50 

x= kl = L 4 J 

The sample variances and covariances are 

4 

*n = 4 E ( x yi ~ *i) 2 
; = 1 

- \((42 - 50) 2 + (52 - 50) 2 + (48 - 50) 2 + (58 - 50) 2 ) = 34 

4 

*22 — 4 ( X ;2 

/ = ! 

= i((4 - 4) 2 + (5 - 4) 2 + (4 - 4) 2 + (3 - 4) 2 ) = .5 

4 

*12 = \ 2 ( x jl ~ *l)( X j2 ~ *2) 

;=i 


= ±((42 - 50) (4 — 4) + (52 - 50) (5 - 4) 


+ (48 - 50) (4 - 4) + (58 - 50) (3 - 4)) = -1.5 


*21 “ *12 


34 -1.5 
-1.5 .5 
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Graphical Techniques 

H 0 iih a . re / mp0rt r t ’ but ^ Uently ne 8 lected > aids in data analysis. Although it is im- 
^ible to simultaneously plot all the measurements made on several variables and 
study the eonfigurations, plots of individual variables and plots of pairs of variables 
can still be very informative. Sophisticated computer programs and display equip- 
ment allow one the luxury of visually examining data in one, two, or three dimen- 

fmm tT Vfr ease ' ° nthe ° ther hand ’ many valuab,e insights can be obtained 
from the data by constructing plots with paper and pencil. Simple, yet elegant and 

tii^on^n* 0415 / 0 " d,s P lay,n 8 data are available in [29], It is good statistical prac- 

er th^it. P f 3 n S ° fvariables and visual| y ins Pect the pattern of association. Consid¬ 
er, then, the following seven pairs of measurements on two variables: 

Variable 1 (*,): 3 4 2 6 8 2 5 

Variable 2 (x 2 ): 5 5.5 4 7 10 5 7.5 

• ™ ese ^‘a are plotted as seven points in two dimensions (each axis represent- 
mg a vanable) , n Figure 1.1 The coordinates of the points are determined by the 

Ztknor" ementS ; ( : ) ' (4 ' 5 ' 5) ’-’ < 5 ’ 7 ' 5 )- resulting two-dimensional 
plot is known as a scatter diagram or scatter plot. 





Also shown in Figure 1.1 are separate plots of the observed values of variable 1 
and the observed values of variable 2, respectively. These plots are called ( marginal) 
dot diagrams . They can be obtained from the original observations or by projecting 
the points in the scatter diagram onto each coordinate axis. 

The information contained in the single-variable dot diagrams can be used to 
calculate the sample means x x and x 2 and the sample variances s x x and s 2 2 - (See Ex¬ 
ercise LI.) The scatter diagram indicates the orientation of the points, and their co¬ 
ordinates can be used to calculate the sample covariance s 12 . In the scatter diagram 
of Figure 1.1, large values of x Y occur with large values of x 2 and small values of X\ 
with small values of x 2 . Hence, 512 will be positive. 

Dot diagrams and scatter plots contain different kinds of information. The in¬ 
formation in the marginal dot diagrams is not sufficient for constructing the scatter 
plot. As an illustration, suppose the data preceding Figure 1.1 had been paired dif¬ 
ferently, so that the measurements on the variables x x and x 2 were as follows: 

Variable 1 (* 1 ): 5 4 6 2 2 8 3 

Variable 2 (x 2 ): 5 5.5 4 7 10 5 7.5 

(We have simply rearranged the values of variable 1.) The scatter and dot diagrams 
for the “new” data are shown in Figure 1.2. Comparing Figures 1.1 and 1.2, we find 
that the marginal dot diagrams are the same, but that the scatter diagrams are decid¬ 
edly different. In Figure 1.2, large values of x x are paired with small values of x 2 and 
small values of x x with large values of x 2 . Consequently, the descriptive statistics for 
the individual variables x x ,x 2 ,s n , and s 22 remain unchanged,but the sample covari¬ 
ance 5 12 , which measures the association between pairs of variables, will now be 
negative. 

The different orientations of the data in Figures 1.1 and 1.2 are not discernible 
from the marginal dot diagrams alone. At the same time, the fact that the marginal 
dot diagrams are the same in the two cases is not immediately apparent from the 
scatter plots. The two types of graphical procedures complement one another; they 
are not competitors. 

The next two examples further illustrate the information that can be conveyed 
by a graphic display. 
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Example 1.3 (The effect of unusual observations on sample correlations) Some fi¬ 
nancial data representing jobs and productivity for the 16 largest publishing firms 
appeared in an article in Forbes magazine on April 30,1990.The data for the pair of 
variables x x — employees (jobs) and x 2 = profits per employee (productivity) are 
graphed in Figure 1.3. We have labeled two “unusual” observations. Dun & Brad- 
street is the largest firm in terms of number of employees, but is “typical” in terms of 
profits per employee. Time Warner has a “typical” number of employees, but com¬ 
paratively small (negative) profits per employee. 



Dun & Bradstreet 


Time Warner 


0 10 20 30 40 50 60 70 80 

Employees (thousands) 


Figure 1.3 Profits per employee 
and number of employees for 16 
publishing firms. 


The sample correlation coefficient computed from the values of x t and x 2 is 

-.39 for all 16 firms 
-.56 for all firms but Dun & Bradstreet 
-.39 for all firms but Time Warner 
-.50 for all firms but Dun & Bradstreet and Time Warner 

It is clear that atypical observations can have a considerable effect on the sample 
correlation coefficient. ■ 



Example 1.4 (A scatter plot for baseball data) In a July 17,1978, article on money in 
sports, Sports Illustrated magazine provided data on x x = player payroll for Nation¬ 
al League East baseball teams. 

We have added data on x 2 = won-lost percentage for 1977. The results are 
given in Table 1.1. 

The scatter plot in Figure 1.4 supports the claim that a championship team can 
be bought. Of course, this cause-effect relationship cannot be substantiated, be¬ 
cause the experiment did not include a random assignment of payrolls. Thus, statis¬ 
tics cannot answer the question: Could the Mets have won with $4 million to spend 
on player salaries? 




14 


Chapter 1 Aspects of Multivariate Analysis 


Table l.l 1977 Salary and Final Record for the National League East 

Team 

X\ = player payroll 

jc 2 = won-lost 
percentage 

Philadelphia Phillies 

3,497,900 

.623 

Pittsburgh Pirates 

2,485,475 

.593 

St. Louis Cardinals 

1,782,875 

.512 

Chicago Cubs 

1,725,450 

.500 

Montreal Expos 

1,645,575 

.463 

New York Mets 

1,469,800 

.395 


*2 

8) 

| .800 

8 

8 . 

o 

I 400 


Figure 1.4 Salaries 
and won-lost 
percentage from 
Table 1.1. 


To construct the scatter plot in Figure 1.4, we have regarded the six paired ob¬ 
servations in Table 1.1 as the coordinates of six points in two-dimensional space. The 
figure allows us to examine visually the grouping of teams with respect to the vari¬ 
ables total payroll and won-lost percentage. 

Example 1.5 (Multiple scatter plots for paper strength measurements) Paper is man¬ 
ufactured in continuous sheets several feet wide.Because of the orientation of fibers 
within the paper, it has a different strength when measured in the direction pro¬ 
duced by the machine than when measured across, or at right angles to, the machine 
direction. Table 1.2 shows the measured values of 

Xi = density (grams/cubic centimeter) 

jc 2 = strength (pounds) in the machine direction 

x 3 = strength (pounds) in the cross direction 

A novel graphic presentation of these data appears in Figure 1.5, page 16. The 
scatter plots are arranged as the off-diagonal elements of a covariance array and 
box plots as the diagonal elements. The latter are on a different scale with this 



0 1.0 2.0 3.0 4.0 

Player payroll in millions of dollars 
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Table 1.2 Paper-Quality Measurements 



3 




Strength (CD) Strength (MD)__^Density 
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PeHSity Strength (MD) Strength (CD) 



Figure 1.5 Scatter plots and boxpiots of paper-quality data from Thble 1.2. 


software so we use only the overall shape to provide information on symmetry 
and Dosslble outliers for each individual characteristic.The scatter plots can be in¬ 
jected for patterns and unusual observations. In Figure 1.5, there is one unusual 
observation: the density of specimen 25. Some of the scatter plots have patterns 
suggesting that there are two separate clumps of observations. 

These scatter plot arrays are further pursued in our discussion of new software 

graphics in the next section. m 

In the general multiresponse situation,/? variables are simultaneously recorded 
on n items. Scatter plots should be made for pairs of important variables and, if the 
task is not too great to warrant the effort, for all pairs. 

Limited as we are to a three-dimensional world, we cannot always picture an 
entire set of data. However, two further geometric representations of the data pro¬ 
vide an important conceptual framework for viewing multivariable statistical meth¬ 
ods. In cases where it is possible to capture the essence of the data in three 
dimensions, these representations can actually be graphed. 
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n Points in p Dimensions (/^-Dimensional Scatter Plot). Consider the natural exten¬ 
sion of the scatter plot to p dimensions, where the p measurements 

(Xjl , Xj2i • • * , Xjp) 

on the y'th item represent the coordinates of a point in /^-dimensional space. The co¬ 
ordinate axes are taken to correspond to the variables, so that the yth point is x } \ 
units along the first axis, x j2 units along the second,..., xj p units along the /?th axis. 
The resulting plot with n points not only will exhibit the overall pattern of variabili¬ 
ty, but also will show similarities (and differences) among the n items. Groupings of 
items will manifest themselves in this representation. 

The next example illustrates a three-dimensional scatter plot. 


Example 1.6 (Looking for lower-dimensional structure) A zoologist obtained mea¬ 
surements on n = 25 lizards known scientifically as Cophosaurus texanus. The 
weight, or mass, is given in grams while the snout-vent length (SVL) and hind limb 
span (HLS) are given in millimeters. The data are displayed in Table 1.3. 

Although there are three size measurements, we can ask whether or not most of 
the variation is primarily restricted to two dimensions or even to one dimension. 

To help answer questions regarding reduced dimensionality, we construct the 
three-dimensional scatter plot in Figure 1.6. Clearly most of the variation is scatter 
about a one-dimensional straight line. Knowing the position on a line along the 
major axes of the cloud of points would be almost as good as knowing the three 
measurements Mass, SVL, and HLS. 

However, this kind of analysis can be misleading if one variable has a much 
larger variance than the others. Consequently, we first calculate the standardized 
values, Zjk - { x jk ~ x k)l'fskk* so the variables contribute equally to the variation 


Table 1.3 Lizard Size Data 

Lizard 

Mass 

SVL 

HLS 

Lizard 

Mass 

SVL 

HLS 

1 

5.526 

59.0 

113.5 

14 

10.067 

73.0 

136.5 

2 

10.401 

75.0 

142.0 

15 

10.091 

73.0 

135.5 

3 

9.213 

69.0 

124.0 

16 

10.888 

77.0 

139.0 

4 

8.953 

67.5 

125.0 

17 

7.610 

61.5 

118.0 

5 

7.063 

62.0 

129.5 

18 

7.733 

66.5 

133.5 

6 

6.610 

62.0 

123.0 

19 

12.015 

79.5 

150.0 

7 

11.273 

74.0 

140.0 

20 

10.049 

74.0 

137.0 

8 

2.447 

47.0 

97.0 

21 

5.149 

59.5 

116.0 

9 

15.493 

86.5 

162.0 

22 

9.158 

68.0 

123.0 

10 

9.004 

69.0 

126.5 

23 

12.132 

75.0 

141.0 

11 

8.199 

70.5 

136.0 

24 

6.978 

66.5 

117.0 

12 

6.601 

64.5 

116.0 

25 

6.890 

63.0 

117.0 

13 

7.622 

67.5 

135.0 






Source: Data courtesy of Kevin E. Bonine. 






Figure 1.6 3D scatter 
plot of lizard data from 
Table 1.3. 


tter plot. Figure 1.7 gives the three-dimensional scatter plot for the stan- 
in the s ^ aria y es . Most of the variation can be explained by a single vanable de- 
dardizedy through the cloud of points. 

tertnineu oy 
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Figure 1.7 3D scatter 
plot of standardized 
lizard data. 1 


v three-dime™ 01 ’ 31 matter plot can often reveal group structure. 

--TTTrLooking for group structure in three dimensions) Referring to Exam- 

•xampl* ''. j resting to see if male and female lizards occupy different parts of the 
P ^VmeS space containing the size data.TTie gender, by row, for the lizard 
Stable 1-3 are 
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Figure 1.8 repeats the scatter plot for the original variables but with males 
marked by solid circles and females by open circles. Clearly, males are typically larg¬ 
er than females. 



Figure 1.8 3D scatter plot of male and female lizards. ■ 

p Points in n Dimensions. The n observations of the p variables can also be re¬ 
garded as p points in /t-dimensional space. Each column of X determines one of the 
points. The t'th column, 


consisting of all n measurements on the zth variable, determines the zth point. 

In Chapter 3, we show how the closeness of points in n dimensions can be relat¬ 
ed to measures of association between the corresponding variables . 

1.4 Data Displays and Pictorial Representations 

The rapid development of powerful personal computers and workstations has led to 
a proliferation of sophisticated statistical software for data analysis and graphics. It 
is often possible, for example, to sit at one’s desk and examine the nature of multidi¬ 
mensional data with clever computer-generated pictures. These pictures are valu¬ 
able aids in understanding data and often prevent many false starts and subsequent 
inferential problems. 

As we shall see in Chapters 8 and 12, there are several techniques that seek to 
represent p-dimensional observations in few dimensions such that the original dis¬ 
tances (or similarities) between pairs of observations are (nearly) preserved. In gen¬ 
eral, if multidimensional observations can be represented in two dimensions, then 
outliers, relationships, and distinguishable groupings can often be discerned by eye. 
We shall discuss and illustrate several methods for displaying multivariate data in 
two dimensions. One good source for more discussion of graphical methods is [11]. 
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Unking Multiple Two-Dimensional Scatter Plots 

One of the more exciting new graphical procedures involves electronically connect¬ 
ing many two-dimensional scatter plots. 


Example 1.8 (Linked scatter plots and brushing) To illustrate linked two-dimensional 
scatter plots, we refer to the paper-quality data in Thble 1.2. These data represent 
measurements on the variables x x = density, x 2 = strength in the machine direction, 
and x 3 = strength in the cross direction. Figure 1.9 shows two-dimensional scatter 
plots for pairs of these variables organized as a 3 X 3 array. For example, the picture 
in the upper left-hand comer of the figure is a scatter plot of the pairs of observations 
(* 1 , * 3 )’ ^ at *1 values are plotted along the horizontal axis, and the x 3 values 
are plotted along the vertical axis.The lower right-hand comer of the figure contains a 
scatter plot of the observations (x 3 , xj. That is,the axes are reversed. Corresponding 
interpretations hold for the other scatter plots in the figure. Notice that the variables 
and their three-digit ranges are indicated in the boxes along the SW-NE diagonal. The 
operation of marking ( selecting ), the obvious outlier in the (x lt x 3 ) scatter plot of 
Figure 1.9 creates Figure 1.10(a), where the outlier is labeled as specimen 25 and the 
same data point is highlighted in all the scatter plots. Specimen 25 also appears to be 
an outlier in the (x x , x 2 ) scatter plot but not in the (x 2t x 3 ) scatter plot.The operation 
of deleting this specimen leads to the modified scatter plots of Figure 1.10(b). 

From Figure 1.10, we notice that some points in,for example, the (x 2l x 3 ) scatter 
plot seem to be disconnected from the others. Selecting these points, using the 
(dashed) rectangle (see page 22), highlights the selected points in all of the other 
scatter plots and leads to the display in Figure 1.11(a). Further checking revealed 
that specimens 16-21, specimen 34, and specimens 38-41 were actually specimens 
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Figure 1.9 Scatter 
plots for the paper- 
quality data of 
Table 1.2. 
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selected and 
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from an older roll of paper that was included in order to have enough plies in the 
cardboard being manufactured. Deleting the outlier and the cases corresponding to 
the older paper and adjusting the ranges of the remaining observations leads to the 
scatter plots in Figure 1.11(b). 

The operation of highlighting points corresponding to a selected range of one of 
the variables is called brushing. Brushing could begin with a rectangle, as in Figure 
1.11(a), but then the brush could be moved to provide a sequence of highlighted 
points. The process can be stopped at any time to provide a snapshot of the current 
situation. ■ 


Scatter plots like those in Example 1.8 are extremely useful aids in data analy¬ 
sis. Another important new graphical technique uses software that allows the data 
analyst to view high-dimensional data as slices of various three-dimensional per¬ 
spectives. This can be done dynamically and continuously until informative views 
are obtained. A comprehensive discussion of dynamic graphical methods is avail¬ 
able in [X]. A strategy for on-line multivariate exploratory graphical analysis, moti¬ 
vated by the need for a routine procedure for searching for structure in multivariate 
data, is given in [32]. 


Example 1.9 (Rotated plots in three dimensions) Four different measurements of 
lumber stiffness are given in Table 4.3, page 186. In Example 4.14, specimen (board) 
16 and possibly specimen (board) 9 are identified as unusual observations. Fig¬ 
ures 1.12(a), (b), and (c) contain perspectives of the stiffness data in the x lt x 2 ,x 3 
space. These views were obtained by continually rotating and turning the three- 
dimensional coordinate axes. Spinning the coordinate axes allows one to get a better 



(a) Outliers clear. (b) Outliers masked. 



(d) Good view of 

(c) Specimen 9 large. jc 2 , jc 3 , x 4 space. 


Figure 1.12 Three-dimensional perspectives for the lumber stiffness data. 
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understanding of the three-dimensional aspects of the data. Figure 1.12(d) gives 
one picture of the stiffness data in jc 2 , x 3 , x 4 space. Notice that Figures 1.12(a) and 
(d) visually confirm specimens 9 and 16 as outliers. Specimen 9 is very large in all 
three coordinates. A counterclock wiselike rotation of the axes in Figure 1.12(a) 
produces Figure 1.12(b), and the two unusual observations are masked in this view. 
A further spinning of the jc 2 , *3 axes gives Figure 1.12(c); one of the outliers (16) is 
now hidden. 

Additional insights can sometimes be gleaned from visual inspection of the 
slowly spinning data. It is this dynamic aspect that statisticians are just beginning to 
understand and exploit. ■ 

Plots like those in Figure 1.12 allow one to identify readily observations that do 
not conform to the rest of the data and that may heavily influence inferences based 
on standard data-generating models. 

Graphs of Growth Curves 

When the height of a young child is measured at each birthday, the points can be 
plotted and then connected by lines to produce a graph. This is an example of a 
growth curve . In general, repeated measurements of the same characteristic on the 
same unit or subject can give rise to a growth curve if an increasing, decreasing, or 
even an increasing followed by a decreasing, pattern is expected. 

Example 1.10 (Arrays of growth curves) The Alaska Fish and Game Department 
monitors grizzly bears with the goal of maintaining a healthy population. Bears are 
shot with a dart to induce sleep and weighed on a scale hanging from a tripod. Mea¬ 
surements of length are taken with a steel tape. Table 1.4 gives the weights (wt) in 
kilograms and lengths (lngth) in centimeters of seven female bears at 2, 3, 4, and 5 
years of age. 

First, for each bear, we plot the weights versus the ages and then connect the 
weights at successive years by straight lines. Tliis gives an approximation to growth 
curve for weight. Figure 1.13 shows the growth curves for all seven bears. The notice¬ 
able exception to a common pattern is the curve for bear 5. Is this an outlier or just 
natural variation in the population? In the field, bears are weighed on a scale that 


Table 1.4 Female Bear Data 

Bear 

Wt2 

Wt3 

Wt4 

Wt5 

Lngth 2 

Lngth 3 

Lngth 4 

Lngth 5 

1 

48 

59 

95 

82 

141 

157 

168 

183 

2 

59 

68 

102 

102 

140 

168 

174 

170 

3 

61 

77 

93 

107 

145 

162 

172 

177 

4 

54 

43 

104 

104 

146 

159 

176 

171 

5 

100 

145 

185 

247 

150 

158 

168 

175 

6 

68 

82 

95 

118 

142 

140 

178 

189 

7 

68 

95 

109 

111 

139 

171 

176 

175 


Source: Data courtesy of H. Roberts. 
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2.0 2.5 3.0 3.5 4.0 4.5 5.0 


Year 


Figure 1.13 Combined 
growth curves for weight 
for seven female grizzly 
bears. 


reads pounds. Further inspection revealed that, in this case, an assistant later failed to 
convert the field readings to kilograms when creating the electronic database. The 
correct weights are (45, 66,84,112) kilograms. 

Because it can be difficult to inspect visually the individual growth curves in a 
combined plot, the individual curves should be replotted in an array where similari¬ 
ties and differences are easily observed. Figure 1.14 gives the array of seven curves 
for weight. Some growth curves look linear and others quadratic. 


Bear 1 Bear 2 Bear 3 Bear 4 



Year Year Year 

Figure M4 Individual growth curves for weight for female grizzly bears. 













Length Lenglh 
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Figure 1.15 gives a growth curve array for length. One bear seemed to get shorter 
from 2 to 3 years old, but the researcher knows that the steel tape measurement o 
length can be thrown off by the bear’s posture when sedated. 


Bear 1 Bear 2 Bear3 



Year Year Year 

l 15 Individual growth curves for length for female grizzly bears. ■ 

figure 1 

We now turn to two popular pictorial representations of multivariate data in 
two dimensions: stars and Chemoff faces. 

Stars 

Suonose each data unit consists of nonnegative observations on p a 2 variables. In 
two dimensions, we can construct circles of a fixed (reference) radius with p equally 
sDaced rays emanating from the center of the circle. The lengths of the rays represent 
the values of the variables. The ends of the rays can be connected with straight lines to 
form a star. Each star represents a multivariate observation, and the stars can e 

grouped according to their (subjective) similarities. 

^ It is often helpful, when constructing the stars, to standardize the observations. 
In this case some of the observations will be negative. The observations can then be 
reexpressed so. that the center of the circle represents the smallest standardized 
observation within the entire data set. 


Example l.l 1 (Utility data as stars) Stars representing the first 5 of the 22 public 
utility firms in Table 12.4, page 688, are shown in Figure 1.16. There are eight vari¬ 
ables; consequently, the stars are distorted octagons 
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Arizona Public Service (I) Boston Edison Co, (2) 



Figure 1.16 Stars for the first five public utilities. 


The observations on all variables were standardized. Among the first five utili¬ 
ties, the smallest standardized observation for any variable was -1.6. Treating this 
value as zero, the variables are plotted on identical scales along eight equiangular 
rays originating from the center of the circle. The variables are ordered in a clock¬ 
wise direction, beginning in the 12 o’clock position. 

At first glance, none of these utilities appears to be similar to any other. However, 
because of the way the stars are constructed, each variable gets equal weight in the vi¬ 
sual impression. If we concentrate on the variables 6 (sales in kilowatt-hour [kWh] use 
per year) and 8 (total fuel costs in cents per kWh), then Boston Edison and Consoli¬ 
dated Edison are similar (small variable 6, large variable 8), and Arizona Public Ser¬ 
vice, Central Louisiana Electric, and Commonwealth Edison are similar (moderate 
variable 6, moderate variable 8). u 

Chernoff Faces 

People react to faces. Chernoff [4] suggested representing p-dimensional observa¬ 
tions as a two-dimensional face whose characteristics (face shape, mouth curvature, 
nose length, eye size, pupil position, and so forth) are determined by the measure¬ 
ments on the p variables. 
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As originally designed, Chernoff faces can handle up to 18 variables. The assign¬ 
ment of variables to facial features is done by the experimenter, and different choic¬ 
es produce different results. Some iteration is usually necessary before satisfactory 
representations are achieved. 

Chernoff faces appear to be most useful for verifying (1) an initial grouping sug¬ 
gested by subject-matter knowledge and intuition or (2) final groupings produced 
by clustering algorithms. 


Example 1.12 (Utility data as Chernoff faces) From the data in Table 12.4, the 22 
public utility companies were represented as Chernoff faces. We have the following 
corre spond ences: 


Variable 


Facial characteristic 

X Y : Fixed-charge coverage 

*—> 

Half-height of face 

X 2 : Rate of return on capital 

*—> 

Face width 

X 3: Cost per kW capacity in place 


Position of center of mouth 

X 4 : Annual load factor 


Slant of eyes 

X s : Peak kWh demand growth from 1974 

<—> 

( height \ 

Eccentricity wjdth J of eyes 

X 6 : Sales (kWh use per year) 


Half-length of eye 

X 7 : Percent nuclear 

<r-> 

Curvature of mouth 

X R : Total fuel costs (cents per kWh) 


Length of nose 


The Chernoff faces are shown in Figure 1.17. We have subjectively grouped 
“similar” faces into seven clusters. If a smaller number of clusters is desired, we 
might combine clusters 5,6,and 7 and,perhaps,clusters2 and 3 to obtain four or five 
clusters. For our assignment of variables to facial features, the firms group largely 
according to geographical location. ■ 


Constructing Chernoff faces is a task that must be done with the aid of a com¬ 
puter. The data are ordinarily standardized within the computer program as part of 
the process for determining the locations, sizes, and orientations of the facial char¬ 
acteristics. With some training, we can use Chernoff faces to communicate similari¬ 
ties or dissimilarities, as the next example indicates. 


Example M3 (Using Chernoff faces to show changes over time) Figure 1.18 illus¬ 
trates an additional use of Chernoff faces. (See [24].) In the figure, the faces are used 
to track the financial well-being of a company over time. As indicated, each facial 
feature represents a single financial indicator, and the longitudinal changes in these 
indicators are thus evident at a glance. ■ 


Cluster 1 
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Figure 1.17 Chernoff faces for 22 public utilities. 
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Figure 1.18 Chernoff faces over time. 


Cluster 7 



>-Time 



30 


Chapt er 1 Aspects of Multivariate Analysis 


Chernoff faces have also been used to display differences in multivariate obser¬ 
vations in two dimensions. For example, the two-dimensional coordinate axes might 
represent latitude and longitude (geographical location), and the faces might repre¬ 
sent multivariate measurements on several U.S. cities. Additional examples of this 
kind are discussed in [30]. 

There are several ingenious ways to picture multivariate data in two dimensions. 
We have described some of them. Further advances are possible and will almost 
certainly take advantage of improved computer graphics. 


1 5 Distance 


Although they may at first appear formidable, most multivariate techniques are based 
upon the simple concept of distance. Straight-line, or Euclidean, distance should be 
familiar. If we consider the point P — (xj, x 2 ) in the plane, the straight-line distance, 
d(0 , P), from P to the origin O - (0, 0) is, according to the Pythagorean theorem, 


d(0,P) = Vx] + xj (1-9) 

The situation is illustrated in Figure 1.19. In general, if the point P has p coordi¬ 
nates so that P - (xj, * 2 , ♦ • - > x p )i the straight-line distance from P to the origin 
O = ( 0 , 0 , is 


d(0 , P) = 


'x\ + x\ + 


(See Chapter 2.) All points (x x , x 2 ,..., x p ) that lie a constant squared distance, such 
as c 2 , from the origin satisfy the equation 


d 2 (0 , P) = x\ + x\ + 


■ + x* = c* 


Because this is the equation of a hypersphere (a circle if p ~ 2), points equidistant 
from the origin lie on a hypersphere. 

The straight-line distance between two arbitrary points P and Q with coordi- 
nates P = (x u x 2 ,-. ■, x p ) and Q = is given by 

d(P,Q) = N/(jci - >i ) 2 + (*2 ~ tt ) 2 + • • + (-* P ~^) 2 (1-12) 


Straight-line, or Euclidean, distance is unsatisfactory for most statistical purpos¬ 
es This is because each coordinate contributes equally to the calculation of Euclid¬ 
ean distance. When the coordinates represent measurements that are subject to 
random fluctuations of differing magnitudes, it is often desirable to weight coordi¬ 
nates subject to a great deal of variability less heavily than those that are not highly 
variable. This suggests a different measure of distance. 

Our purpose now is to develop a “statistical” distance that accounts for differ¬ 
ences in variation and, in due course, the presence of correlation. Because our 



Figure 1.19 Distance given 
by the Pythagorean theorem. 
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choice will depend upon the sample variances and covariances at this point we use 
statical distance to distinguish it from ordinary EucHdean dEce It is 
tisttcal distance that is fundamental to multivariate analysis 

scatter ^From^e “ ^ ‘If Set ° f ° bservatlons graphed as the /,-dimensional 
a point P = , hCSe ’ We sba " ^struct a measure of distance from the origin to 
point r (x, x 2 ,..., Xp ). In our arguments, the coordinates (x ,, x, x) of P 

having mS I 2 rocT t he We ^ ^ ° f measure “ two variables each 

vary^dependenUv^ tJl ** 3SSUme that the *1 measurements 

in t K Pdy f he * 2 measurements . 1 In addition, assume that the variability 
in the x, measurements is larger than the variability in the * 2 measurements. A scatte^ 
plot of the data would look something like the one pictured in Figure 12 ? 


*1 

Figure 1.20 A scatter plot with 
greater variability in the X\ direction 
than in the x 2 direction. 

Glancing at Figure 1.20, we see that values which are a given deviation from the 

tant'from fhe o "°‘ 35 “ sur P risin g" or “unusual” as are values equidis- 

r Hir.T h 8 m u he X2 dlrection This is because the inherent variability in the 
x, direction is greater than the variability in the , 2 direction. Consequently . al x' 
coordinates (in absolute value) are not as unexpected as large x 2 coordinates I 
seems reasonabJe, then, to weight an x 2 coordinate more heavily than an x, coordi 
nate of the same value when computing the “distance” to the origin 

tion t0 pr ° Ce< - d is t0 divide each coor dinat e by the sample standard devia¬ 
ted” coordinates^* 1 -^ 1V1 / 1 -\/~ y s . tandard Rations, we have the “standard- 

are now orfan emiaI f~ *' /V *\' and The standardized coordinates 

are now on an equal footing with one another. After taking the differences in vari- 

Sus r s tSSd We t deter 7 i r diStanCe USiDg thC St3ndard Eu didean formula, 
bus, a statistical distance of the pomt P = ( Xj , x 2 ) from the origin O = 10 01 can 

be computed from .ts standardized coordinates VV^andxfL * 2 /v|fas 

d(0,P) = V(xl) 2 + (x 2 *) 2 



accuracy from die x, i^asuremtnt^anTvfcl ** m ^ SmementS cannot be Predicted with any 
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Comparing (1-13) with (1-9), we see that the difference between the two expres¬ 
sions is due to the weights k, = 1 /s n and k 2 = l/s 22 attached to x^ and x 2 m (1-13). 
Note that if the sample variances are the same, k\ = k 2 , then and x 2 will receive 
the same weight. In cases where the weights are the same, it is convenient to ignore the 
common divisor and use the usual Euclidean distance formula. In other words, if 
the variability in the-x, direction is the same as the variability in the x 2 direction, 
and the xj values vary independently of the x 2 values, Euclidean distance is 

appropriate. . , 

Using (1-13), we see that all points which have coordinates (x l9 x 2 ) and are a 

constant squared’distance c 2 from the origin must satisfy 



$11 S 2 2 


(1-14) 


Equation (1-14) is the equation of an ellipse centered at the origin whose major and 
minor axes coincide with the coordinate axes. That is, the statistical distance in 
(1-13) has an ellipse as the locus of all points a constant distance from the origin. 
This general case is shown in Figure 1.21. 



Example 1.14 (Calculating a statistical distance) A set of paired measurements 
(xi x 2 ) on two variables yields x, = x 2 = 0, s n = 4, and s 22 = 1. Suppose the x, 
measurements are unrelated to the x 2 measurements; that is, measurements within a 
pair vary independently of one another. Since the sample variances are unequal, we 
measure the square of the distance of an arbitrary point P = (x lt x 2 ) to the ongin 
O = (0, 0) by 


d 2 (0,P) 



All points (x,, x 2 ) that are a constant distance 1 from the origin satisfy the equation 



The coordinates of some points a unit distance from the origin are presented in the 
following table: 
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A plot of the equation ; x\/A -h jc|/ 1 = 1 is an ellipse centered at (0,0) whose 
major axis lies along the x x coordinate axis and whose minor axis lies along the x 2 
coordinate axis. The half-lengths of these major and minor axes are V4 = 2 and 
VT = 1 , respectively. The ellipse of unit distance is plotted in Figure 1.22. All points 
on the ellipse are regarded as being the same statistical distance from the origin—in 
this case, a distance of 1. ■ 



The expression in (1-13) can be generalized to accommodate the calculation of 
statistical distance from an arbitrary point P = {x Xi x 2 ) to any fixed point 
Q ~ (yi, yz)> If we assume that the coordinate variables vary independently of one 
another, the distance from P to Q is given by 


d{P,Q) 



(*2 ~ - V ;) 2 
*22 


(1-15) 


The extension of this statistical distance to more than two dimensions is 
straightforward. Let the points P and Q have p coordinates such that 
P = (*j > * 2 , • • * > X P ) and Q ~ (yi, yz> * • •, y P )■ Suppose Q is a fixed point [it may be 
the origin O = (0,0,, 0)] and the coordinate variables vary independently of one 
another. Let s n , s 2 2 , be sample variances constructed from n measurements 

on xi , x 2 ,..., x p , respectively. Then the statistical distance from P to Q is 



, a cnects of Multivariate Analysis 
rb apt et ^ 

34 ah ooints P that are a constant squared distance from Q lie on a hyperelhpsoid 

STS who- major and nJr axes am parallel ro rh« eoordmar. area We 

note the following: 

j The distance of P to the origin O is obtained by setting tt = K ~ y P 0 

If sn =\ 22 " = • • • = s pp , the Euclidean distance formula in (1-12) is appropriate. 

_ ,• tan „ in n.i6) still does not include most of the important cases we shall 

^ e .e ?ec?us“ of the assumption of independent coordinates. The scatter plot 
en< * >ul ; 23 depicts a two-dimensional situation in which the x t measurements do 
in l '6 ur • Hpntlv of the xi measurements. In fact, the coordinates of the pairs 
no* Var i y lSra^tendency Jo S "Lll together, and the sample correlat on 

^efficient is positive. Moreover, the variability in the x 2 direction is larger t an e 
^^^^‘me^rfSmeasure of distance when the variability in the x t direc- 

inal coordinate system g ^ terms of the new axes looks very much like 

(You ma, wish to tun, the book ,o place the I, and 5, axes ,n 

their cusio y £ and measure distance as m Equation (M3). lHat is, 

to rttl" axes, we detoe the distattce „ot» the pom. 
y i (xu x 2 ) to the origin 0= (0,0) as 


d(0, P ) 



(1-17) 


* h ere s„ and ? 22 denote the sample variances computed with the x, and x 2 

measurements. 




Figure L23 A scatter plot for 
positively correlated 
measurements and a rotated 
coordinate system. 
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The relation between the original coordinates (jq, x 2 ) and the rotated coordi¬ 
nates (*i, x 2 ) is provided by 

Xj - Xi cos(0) + x 2 sin (&) 

(1-18) 

x 2 = -jqsin(0) -I- x 2 cos(0) 

Given the relations in (1-18), we can formally substitute for and x 2 in (1-17) 
and express the distance in terms of the original coordinates. 

After some straightforward algebraic manipulations, the distance from 
P = (Tj, x 2 ) to the origin O = (0, 0) can be written in terms of the original coordi¬ 
nates jq and x 2 of P as 

d(0, P) = Vfl! + 2 a X 2XyX 2 + U 2 2*2 (1-19) 

where the a’s are numbers such that the distance is nonnegative for all possible val¬ 
ues of Xy and x 2 . Here a n ,a 12 , and a 22 are determined by the angle 6, and r n , r 12 , 
and s 22 calculated from the original data. 2 The particular forms for a l2 y and a 12 
are not important at this point. What is important is the appearance of the cross- 
product term 2a 12 jiq;t2 necessitated by the nonzero correlation r 12 . 

Equation (1-19) can be compared with (1-13). The expression in (1-13) can be 
regarded as a special case of (1-19) with a n = l/syy> a 22 = 1/^22* and c2 12 = 0. 

In general, the statistical distance of the point P = (jq, x 2 ) from the fixed, point 
Q - (jD^) l° r situations in which the variables are correlated has the general 
form 

d(P , Q) = vQx, - yj) 2 + 2 a 12 (x! - yi){x 2 ~ yi) + a 12 (x 2 - yi) 2 (1-20) 

and can always be computed once a 11 ,a 12 , and a 22 are known. In addition, the coor¬ 
dinates of all points P = (oq, jc 2 ) that are a constant squared distance c 2 from Q 
satisfy 

flnO*! - yi) 2 + 2fli 2 (x! - y t )(*2 ~ ») + a 22 (x 2 - Ji) 2 = c 2 (1-21) 

By definition, this is the equation of an ellipse centered at Q. The graph of such an 
equation is displayed in Figure 1.24. The major (long) and minor (short) axes are in¬ 
dicated. They are parallel to the jc 2 and x 2 axes. For the choice of ay !, a 12 , and a 2 2 * n 
footnote 2, the Xy and x 2 axes are at an angle 0 with respect to the x Y and x 2 axes. 

The generalization of the distance formulas of (T19) and (1-20) to p dimen¬ 
sions is straightforward. Let P = (jq , x 2> ..., x p ) be a point whose coordinates 
represent variables that are correlated and subject to inherent variability. Let 


^Specifically, 


cos 2 (0) 

sin 2 (0) 

11 cos 2 (0)5n + 2$in(0) cos(0)j 12 + sin 2 (0)j 22 

cos 2 (0)f 22 - 2sin(0)cos(0)s l2 + sin 2 (0)s n 

sin 2 (0) 

cos 2 (B ) 

22 cos 2 (0)j n + 2 sin(0) cos(0)j 12 + sin 2 (0)j 22 

cos 2 ( 0 )j 2 2 - 2sin(0)cos(0)ji 2 + sin 2 (0)Jn 

and 


cos(0) sin(0) 

sin(0) cos(0) 

12 cos 2 (0)q 1 + 2$in(0)cos(0)5 l2 + sin 2 (0)s2 2 

cos 2 (0)s 22 - 2sin(0)cos(0)s 12 + sin 2 (fi)jji 
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Figure 1.24 Ellipse of points 
a constant distance from the 
point Q. 



n = ( oo 0) denote the origin, and let Q - ■•••J'p) be a specified 

fixed point. Then the distances from P to O and from P to Q have the general 

forms __—--- 

d(OP) = VV? + *22^ + - + *pA + tollXl*! + 2*13*1*3 + ■" + ^P-^P X P-1 X P 


d(P,Q) 


______ ____~ 

/i^T- y,) 2 + *2 2 (*2 - yi) 2 + + “pp(Xp - ^) 2 + 2fll2(j:i ■ y ' } / X2 ~ * 

I + 2 a }3 (x\ - yi)(*3- ft) + ■■■ + 2a P -i,p( x p-i y P -i)( x P yp)} 


(1-23) 


where the a’s are numbers such that the distances are always nonnegative 3 

We note that the distances in (1-22) and (1-23) are completely determmed by 
the coefficients (weights) a ik , i = 1,2.....** = lA-.P-^esecoefftctentscan 
be set out in the rectangular array 


au 

an " 

■ «i P ~ 

an 

^22 ' 

■ ■ °2p (1-24) 

_&] p 

^2 p 

a pp- 


. the a Ps with i * k are displayed twice, since they are multiplied by 2 in the 
distance formulas. Consequently, the entries in this array specify the distance func¬ 
tions The a ik ’s cannot be arbitrary numbers; they must be such that the comput 
distance is nonnegative for every pair of points.(SeeExerc,se 1 JO.) 

Contours of constant distances computed from (1-22) and (1 Li) a 
hyperellipsoids. A hyperellipsoid resembles a football when p = 3; it is impossible 
to visualize in more than three dimensions. 


w, fllefbraic expressions for the squares of the distances in (1-22) and (1-23) are known as .qua¬ 
dratic forms in particular, positive definite quadratic forms. It is ^ d ,splay these quadra ic 

forms in a simpler manner using matrix algebra; we shall do so in Sect,on 2.3 of Chapter 2. 
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* l Figure 1.25 A cluster of points 
relative to a point P and the origin. 

The need to consider statistical rather than Euclidean distance is illustrated 
heuristically in Figure 1.25, Figure 1,25 depicts a cluster of points whose center of 
gravity (sample mean) is indicated by the point Q. Consider the Euclidean distances 
from the point Q to the point P and the origin O.The Euclidean distance from Q to 
P is larger than the Euclidean distance from Q to O. However, P appears to be more 
like the points in the cluster than does the origin. If we take into account the vari¬ 
ability of the points in the cluster and measure distance by the statistical distance in 
(1-20), then Q will be closer to P than to O . This result seems reasonable, given the 
nature of the scatter. 

Other measures of distance can be advanced. (See Exercise 1.12.) At times, it is 
useful to consider distances that are not related to circles or ellipses. Any distance 
measure d(P, Q ) between two points P and Q is valid provided that it satisfies the 
following properties, where R is any other intermediate point; 

d(P,Q) = d(Q f P) 
d(P,Q) > 0 if P * Q 

d(P y Q) = OifF = Q ( ' 

d(P,Q) < d(P,R) + d(R y Q) (triangle inequality) 

1.6 Final Comments 

We have attempted to motivate the study of multivariate analysis and to provide 
you with some rudimentary, but important, methods for organizing, summarizing, 
and displaying data. In addition, a general concept of distance has been introduced 
that will be used repeatedly in later chapters. 

Exercises _ 

1.1. Consider the seven pairs of measurements (jq, x 2 ) plotted in Figure 1.1: 

jq I 3 4 26 825 

* 2 5 5.5 4 7 10 5 75~ 

Calculate the sample means Xi and x 2 , the sample variances s u and s 22 > and the sample 
covariance s j 2 . 
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I 2 A morning newspaper lists the following used-car prices for a foreign compact with age 
Xl measured in years and selling price x 2 measured in thousands of dollars: 


*1 

1 2 

3 

3 

4 

5 

6 

8 

9 

11 

*2 

18.95 19.00 

17.95 

15.54 

14.00 

12.95 

8.94 

7.49 

6.00 

3.99 


(a) Construct a scatter plot of the data and marginal dot diagrams. 

(b) Infer the sign of the sample covariance s l2 from the scatter plot. 

(c) Compute the sample means x x and x 2 and the sample variances su and $22* Com¬ 
pute the sample covariance s l2 and the sample correlation coefficient r l2 . Interpret 
these quantities. 

(d) Display the sample mean array x, the sample variance-covariance array S„, and the 
sample correlation array R using (1-8). 

I 3 The following are five measurements on the variables x^x 2> and *3. 


*1 

9 

2 

6 

5 

8 

*2 

12 

8 

6 

4 

10 

*3 

3 

4 

0 

2 

1 


Find the arrays x, S n , and R. 

I 4. The world’s 10 largest companies yield the following data: 



(a) Plot the scatter diagram and marginal dot diagrams for variables X\ and x 2 . Com¬ 
ment on the appearance of the diagrams. 

(b) Compute x u x 2f sn, *22, *12> and r 12 . Interpret r 12 . 

1.5. Use the data in Exercise 1.4. 

(a) Plot the scatter diagrams and dot diagrams for (x 2 , x 3 ) and (*1, *3)- Comment on 
the patterns. 

(b) Compute the x, S„, and R arrays for (xj, x 2 ,x } ). 
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1.6. The data in Table 1.5 are 42 measurements on air-pollution variables recorded at 12:00 
noon in the Los Angeles area on different days. (See also the air-pollution data on the 
web at www.prenhall.com/statistics.) 

(a) Plot the marginal dot diagrams for all the variables. 

(b) Construct the x, S n , and R arrays, and interpret the entries in R. 



Source: Data courtesy of Professor G. C. Tiao. 
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1.7. You are given the following n = 3 observations on p = 2 variables: 

Variable 1: — 2 x 2 \ — 'S *31 = ^ 

Variable 2: x i2 = 1 *22 = 2 *32 ~ 4 

(a) Plot the pairs of observations in the two-dimensional “variable space.” That is, con¬ 
struct a two-dimensional scatter plot of the data. 

(b) Plot the data as two points in the three-dimensional “item space. 

I 8 Evaluate the distance of the point P = (-1, -1) to the point Q = (1,0) using the Eu¬ 
clidean distance formula in (1-12) with p = 2 and using the statistical distance m (1-20) 

with a j, = 1/3, an = 4/27, and a i 2 = V 9 - Sketch the focus of points that are a con¬ 

stant squared statistical distance 1 from the point Q. 

1.9, Consider the following eight pairs of measurements on two variables x, and x 2 : 


*1 

-6 

-3 

-2 

1 

2 5 

6 

8 

*2 

-2 

-3 

1 

-1 

2 1 

5 

3 


(a) Plot the data as a scatter diagram, and compute *i 1 , s 22 , and s, 2 . ^ 

fbi Using (1-18), calculate the corresponding measurements on variables xi and Xj, as- 
' ’ , lim ing that the original coordinate axes are rotated through an angle of 6 - 26 
[given cos (26°) = .899 and sin (26°) = .438], 

(c) Using the x, and x 2 measurements from (b), compute the sample variances j n 

andT 2 2- , 

fdt Consider the new pair of measurements (x lt x 2 ) = (4, -2). Transform these to 
measurements on x, and x 2 using (1-18), and calculate the distance d(0, P) of the 
new point P = (x,, x 2 ) bom the origin O = (0,0) usmg (1-17). 

Note: You will need vi i and s 22 from (c). 

(e) Calculate the distance from P = (4, -2) to the origin O = (0,0) using (1-19) and 
the expressions for aj i, a 22 , and a l2 in footnote 2. 

Note: You will need .s, i, s 22 , and ,v 12 from (a). , .. ~ 

Compare the distance calculated here with the distance calculated usmg the x, and x 2 
values in (d). (Within rounding error, the numbers should be the same.) 

1.10. Are the following distance functions valid for distance from the origin? Explain. 

(a) x j + 4x] + xjx 2 = (distance) 2 

(b) xj - 2x1 = (distance) 2 

I 11 Verify that distance defined by (1-20) with a„ = 4, a 22 = 1, anda 12 = -1 satisfies the 
' ' first three conditions in (1-25). (The triangle inequality is more difficult to verify.) 

1.12. Define the distance from the point P = (xj, x 2 ) to the origin O = (0,0) as 

d(0,P) = max(|xi |, |x 2 |) 

(a) Compute the distance from P = (-3,4) to the origin. 

(b) Plot the locus of points whose squared distance from the origin is 1. 

(c) Generalize the foregoing distance expression to points in p dimensions. 

I 13 A large city has major roads laid out in a grid pattern, as indicated in the following dia- 
g ram 8 Stree y .s 1 through 5 run north-south (NS), and streets A through E ^ east-west 
(EW). Suppose there are retail stores located at intersections (A, 2), (£, 3 ), ana (C, o;. 
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Assume the distance along a street between two intersections in either the NS or EW di¬ 
rection is 1 unit. Define the distance between any two intersections (points) on the grid 
to be the “city block” distance. [For example, the distance between intersections (Z), 1) 
and (C, 2), which we might call d((D, 1), (C, 2)), is given by d((D, 1), (C, 2)) 
= d((D y 1), (D, 2)) + d((D, 2), (C, 2)) =1 + 1=2. Also, d((D, 1), (C, 2)) = 
^((Z) > 1),(C,1))+^((C,1),(C,2))=1 + 1= 2.] 

1 2 3 4 5 

A 

B 

C 

D 

E 

Locate a supply facility (warehouse) at an intersection such that the sum of the dis¬ 
tances from the warehouse to the three retail stores is minimized. 

The following exercises contain fairly extensive data sets. A computer may be necessary for 
the required calculations. 

1 . 14 - Table 1.6 contains some of the raw data discussed in Section 1.2. (See also the multiple- 
sclerosis data on the web at www.prenhall.com/statistics.) Two different visual stimuli 
(51 and 52) produced responses in both the left eye (L) and the right eye (R) of sub¬ 
jects in the study groups. The values recorded in the table include xj (subject’s age); x 2 
(total response of both eyes to stimulus 51, that is, 51 L + 51/?); x$ (difference between 
responses of eyes to stimulus 51, 151L — 51f? |); and so forth. 

(a) Plot the two-dimensional scatter diagram for the variables x 2 and x 4 for the 
multiple-sclerosis group. Comment on the appearance of the diagram. 

(b) Compute the x, S„, and R arrays for the non-multiple-sclerosis and multiple- 
sclerosis groups separately. 

1.15. Some of the 98 measurements described in Section 1.2 are listed in Table 1.7 (See also 
the radiotherapy data on the web at www.prenhall.com/statistics.) The data consist of av¬ 
erage ratings over the course of treatment for patients undergoing radiotherapy. Vari¬ 
ables measured include (number of symptoms, such as sore throat or nausea); x 2 
(amount of activity, on a 1-5 scale); jc 3 (amount of sleep, on a 1-5 scale); x 4 (amount of 
food consumed, on a 1-3 scale); jc 5 (appetite, on a 1-5 scale); and jc 6 (skin reaction, on a 
0-3 scale). 

(a) Construct the two-dimensional scatter plot for variables x 2 and jc 3 and the marginal 
dot diagrams (or histograms). Do there appear to be any errors in the jc 3 data? 

(b) Compute the x, S„, and R arrays. Interpret the pairwise correlations. 

1.16. At the start of a study to determine whether exercise or dietary supplements would slow 
bone loss in older women, an investigator measured the mineral content of bones by 
photon absorptiometry. Measurements were recorded for three bones on the dominant 
and nondominant sides and are shown in Table 1.8. (See also the mineral-content data 
on the web at www.prenhall.com/statistics.) 

Compute the x, S„, and R arrays. Interpret the pairwise correlations. 





42 Chapter 1 Aspects of Multivariate Analysis 


Table 1.6 Multiple-Sclerosis Data 

Non-Multiple-Sclerosis Group Data 

Subject 

number 

*i 

(Age) 

*2 

(51Z, + SIR) 

*3 

|51E - $lrt| 

*4 

(S2L + S2R) 

*5 

\S2L - S2RI 

1 

18 

~ 152.0 

1.6 

198.4 

.0 

2 

19 

138.0 

.4 

180.8 

1.6 

3 

20 

144.0 

.0 

186.4 

.8 

4 

20 

143.6 

3.2 

194.8 

.0 

5 

20 

148.8 

.0 

217.6 

;0 

65 

67 

154.4 

2.4 

205.2 

6.0 

66 

69 

171.2 

1.6 

210.4 

.8 

67 

73 

157.2 

.4 

204.8 

.0 

68 

74 

175.2 

5.6 

235.6 

.4 

69 

79 

155.0 

1.4 

204.4 

.0 

Multiple- 

Sclerosis Group Data 




Subject 

number 

*1 

*2 

*3 

*4 

*5 

1 

23 

148.0 

.8 

205.4 

.6 

2 

25 

195.2 

3.2 

262.8 

.4 

3 

25 

158.0 

8.0 

209.8 

12.2 

4 

28 

134.4 

.0 

198.4 

3.2 

5 

29 

190.2 

14.2 

243.8 

10.6 

25 

57 

165.6 

16.8 

229.2 

15.6 

26 

58 

238.4 

8.0 

304.4 

6.0 

27 

58 

164.0 

.8 

216.8 

.8 

28 

58 

169.8 

.0 

219.2 

1.6 

29 

59 

199.8 

4.6 

250.2 

1.0 

Source: Data courtesy of Dr. G. G. Celesia. 


Table 1.7 Radiotherapy Data 

*i 

Symptoms 

*2 

Activity 

*3 

Sleep 

*4 

Eat 

*5 

Appetite 

*6 

Skin reaction 

.889 

1.389 

1.555 

2.222 

1.945 

1.000 

2.813 

1.437 

.999 

2.312 

2.312 

2.000 

1.454 

1.091 

2.364 

2.455 


3.000 

.294 

.941 




1.000 

2.727 

2.545 

2.819 

2.727 

4.091 

.000 

4.100 





2.000 

.125 


1.437 

1.875 

1.563 

.000 

6.231 

2.769 

1.462 

2.385 


2.000 

3.000 

1.455 


2.273 

3.272 

2.000 

.889 





2.000 


Source: Data courtesy of Mrs. Annette Tealey, R.N, Values of x 2 and less than 1.0 are due to errors 
in the data-collection process. Rows containing values of x 2 and x 3 less than 1.0 may be omitted. 
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tra^k record. iff n ^ “ S K eC,, ° n 12 are llSted in TabIe 1A ( See als ° the national- 
Web at www Prenha]].co m/statistics.) He national track 
n ;“ rds fo , r w ° men in 54 countries can be examined for the relationships among the run- 

cTeffk en a?vT te T *' t ^ * arrayiNotice the magnitudes of the correlation 
coefficients as you go from the shorter (100-meter) to the longer (marathon) running 

distances. Interpret these pairwise correlations. ' J g 

U8 ‘ oeTsemnd^rl 01131 ^ rwnJs j<> r *°™ninThble 1.9 tospeeds measured in meters 

/ rec T, Speed for the 100 ~ m dash for Argentinian women is 

and mirafi? ~ 8 643 m/seC Notlce that ^ records for the 800-m, 1500-m, 3000-m 
and marathon runs are measured in minutes. The marathon is 26.2 miles or 42 195 

coefficients^* ° mpUt f tbe *’ S "’ and R arra >' s - Notice the magnitudes of the correlation 
frtemret lhec? 80 * 0 " 1 the , shorter ( 10 ° m ) <° the longer (marathon) running distances, 
in Exercise 1 j^ pairW1Se COrre atIons ~ Compare your results with the results you obtained 

U9 daTalnlhbleTsand thUh d b °* pIot displays of F ‘Sure 1.5 for (a) the mineral-content 
data in lable 1.8 and (b) the national-track-records data in Table 1.9. 
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Table 1.9 National Track Records for Women 


Country 

100 m 

(s) 

200 m 

(s) 

400 m 

(s) 

800 m 
(min) 

1500 m 
(min) 

3000 m 
(min) 

Marathon 

(min) 

Argentina 

11.57 

22.94 

52.50 

2.05 

4.25 

9.19 

150.32 

Australia 

11.12 

-22.23 

48.63 

1.98 

4.02 

8.63 

143.51 

Austria 

11.15 

22.70 

50.62 

1.94 

4.05 

8.78 

154.35 

Belgium 

11.14 

22.48 

51.45 

1.97 

4.08 

8.82 

143.05 

Bermuda 

11.46 

23.05 

53.30 

2.07 

4.29 

9.81 

174.18 

Brazil 

11.17 

22.60 

50.62 

1.97 

4.17 

9.04 

147.41 

Canada 

10.98 

22.62 

49.91 

1.97 

4.00 

8.54 

148.36 

Chile 

11.65 

23.84 

53.68 

2.00 

4.22 

9.26 

152.23 

China 

10.79 

22.01 

49.81 

1.93 

3.84 

8.10 

139.39 

Columbia 

11.31 

22.92 

49.64 

2.04 

4.34 

9.37 

155.19 

Cook Islands 

12.52 

25.91 

61.65 

2.28 

4.82 

11.10 

212.33 

Costa Rica 

11.72 

23.92 

52.57 

2.10 

4.52 

9.84 

164.33 

Czech Republic 

11.09 

21.97 

47.99 

1.89 

4.03 

8.87 

145.19 

Denmark 

11.42 

23.36 

52.92 

2.02 

4.12 

8.71 

149.34 

Dominican Republic 

11.63 

23.91 

53.02 

2.09 

4.54 

9.89 

166.46 

Finland 

11.13 

22.39 

50.14 

2.01 

4.10 

8.69 

148.00 

France 

10.73 

21.99 

48.25 

1.94 

4.03 

8.64 

148.27 

Germany 

10.81 

21.71 

47.60 

1.92 

3.96 

8.51 

141.45 

Great Britain 

11.10 

22.10 

49.43 

1.94 

3.97 

8.37 

135.25 

Greece 

10.83 

22.67 

50.56 

2.00 

4.09 

8.96 

153.40 

Guatemala 

11.92 

24.50 

55.64 

2.15 

4.48 

9.71 

171.33 

Hungary 

11.41 

23.06 

51.50 

1.99 

4.02 

8.55 

148.50 

India 

11.56 

23.86 

55.08 

2.10 

4.36 

9.50 

154.29 

Indonesia 

11.38 

22.82 

51.05 

2.00 

4.10 

9.11 

158.10 

Ireland 

11.43 

23.02 

51.07 

2.01 

3.98 

8.36 

142.23 

Israel 

11.45 

23.15 

52.06 

2.07 

4.24 

9.33 

156.36 

Italy 

11.14 

22.60 

51.31 

1.96 

3.98 

8.59 

143.47 

Japan 

11.36 

23.33 

51.93 

2.01 

4.16 

8.74 

139.41 

Kenya 

11.62 

23.37 

51.56 

' 1.97 

3.96 

8.39 

138.47 

Korea, South 

11.49 

23.80 

53.67 

2.09 

4.24 

9.01 

146.12 

Korea, North 

11.80 

25.10 

56.23 

1.97 

4.25 

8.96 

145.31 

Luxembourg 

11.76 

23.96 

56:07 

2.07 

4.35 

9.21 

149.23 

Malaysia 

11.50 

23.37 

52.56 

2.12 

439 

9.31 

169.28 

Mauritius 

11.72 

23.83 

54.62 

2.06 

4.33 

9.24 

167.09 

Mexico 

11.09 

23.13 

48.89 

2.02 

4.19 

8.89 

144.06 

Myanmar(Burma) 

11.66 

23.69 

52.96 

2.03 

4.20 

9.08 

158.42 

Netherlands 

11.08 

22.81 

51.35 

1.93 

4.06 

8.57 

143.43 

New Zealand 

11.32 

23.13 

51.60 

1.97 

4.10 

8.76 

146.46 

Norway 

11.41 

23.31 

52.45 

2.03 

4.01 

8.53 

141.06 

Papua New Guinea 

11.96 

24.68 

55.18 

2.24 

4.62 

10.21 

221.14 

Philippines 

11.28 

23.35 

54.75 

2.12 

4.41 

9.81 

165.48 

Poland 

10.93 

22.13 

49.28 

1.95 

3.99 

8.53 

144.18 

Portugal 

11.30 

22.88 

51.92 

1.98 

3.96 

8.50 

143.29 

Romania 

11.30 

22.35 

49.88 

1.92 

3.90 

8.36 

142.50 

Russia 

10.77 

21.87 

49.11 

1.91 

3.87 

8.38 

141.31 

Samoa 

12.38 

25.45 

56.32 

2.29 

5.42 

13.12 

191.58 


(continues) 
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Country 

100 m 

(s) 

200 m 
(S) 

400 m 

(s) 

800m 

(min) 

1500 m 
(min) 

3000 m 
(min) 

Marathon 

(min) 

Singapore 

12.13 

24.54 

55.08 

2.12 

4.52 

9.94 

154.41 

Spain 

11.06 

22.38 

49.67 

1.96 

4.01 

8.48 

146.51 

Sweden 

11.16 

22.82 

51.69 

1.99 

4.09 

8.81 

150.39 

Switzerland 

11.34 

22.88 

51.32 

1.98 

3.97 

8.60 

145.51 

Taiwan 

11.22 

22.56 

52.74 

2.08 

4.38 

9.63 

159.53 

. Thailand 

11.33 

23.30 

52.60 

2.06 

4.38 

10.07 

162.39 

Tlirkey 

11.25 

22.71 

53.15 

2.01 

3.92 

8.53 

151.43 

U.S.A. 

10.49 

21.34 

48.83 

1.94 

3.95 

8.43 

141.16 


Source: IAAF/ATFS Track and Field Handbook for Helsinki 2005 (courtesy of Ottavio Castellini). 


1 . 20 . Refer to the bankruptcy data in Table 11.4, page 657, and on the following website 
www.prenhall.com/statistics. Using appropriate computer software, 

(a) View the entire data set in x ]> x 2 , x 3 space. Rotate the coordinate axes in various 
directions. Check for unusual observations. 

(b) Highlight the set of points corresponding to the bankrupt firms. Examine various 
three-dimensional perspectives. Are there some orientations of three-dimensional 
space for which the bankrupt firms can be distinguished from the nonbankrupt 
firms? Are there observations in each of the two groups that are likely to have a sig¬ 
nificant impact on any rule developed to classify firms based on the sample means, 
variances, and covariances calculated from these data? (See Exercise 11.24.) 

1.21. Refer to the milk transportation-cost data in Table 6.10, page 345, and on the web at 
www.prenhall.com/statistics. Using appropriate computer software, 

(a) View the entire data set in three dimensions. Rotate the coordinate axes in various 
directions. Check for unusual observations. 

(b) Highlight the set of points corresponding to gasoline trucks. Do any of the gasoline- 
truck points appear to be multivariate outliers? (See Exercise 6.17.) Are there some 
orientations of x Y , x 2 , x 3 space for which the set of points representing gasoline 
trucks can be readily distinguished from the set of points representing diesel trucks? 

1.22. Refer to the oxygen-consumption data in Table 6.12, page 348, and on the web at 
www.prenhall.com/statistics. Using appropriate computer software, 

(a) View the entire data set in three dimensions employing various combinations of 
three variables to represent the coordinate axes. Begin with the x } , x 2 , x 3 space. 

(b) Check this data set for outliers. 

1.23. Using the data in Table 11.9, page 666, and on the web at www.prenhall.com/ 
statistics, represent the cereals in each of the following ways. 

(a) Stars. 

(b) Chemoff faces. (Experiment with the assignment of variables to facial characteristics.) 

1 . 24 . Using the utility data in Table 12.4, page 688, and on the web at www.prenhall. 
com/statistics, represent the public utility companies as Chemoff faces with assign¬ 
ments of variables to facial characteristics different from those considered in Exam¬ 
ple 1.12. Compare your faces with the faces in Figure 1.17. Are different groupings 
indicated? 
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1.25. Using the data in Table 12.4 and on the web at www.prenhall.com/statistics, represent the 
22 public utility companies as stars. Visually group the companies into four or five 
clusters. 

1.26. The data inThble 1.10 (see the bull data on the web at www.prenhall.com/statistics) are 
the measured characteristics of 76 young (less than two years old) bulls sold at auction. 
Also included in the table are the selling prices (SalePr) of these bulls. The column head¬ 
ings (variables) are defined as follows: 


! 1 Angus 
5 Hereford 
8 Simental 


YrHgt = Yearling height at 
shoulder (inches) 


FtFrBody = Fat free body PrctFFB = Percent fat-free 

(pounds) body 

Frame = Scale from 1 (small) BkFat = Back fat 
to 8 (large) (inches) 


SaleHt = Sale height at SaleWt = Sale weight 

shoulder (inches) (pounds) 

(a) Compute the x, S„, and R arrays. Interpret the pairwise correlations. Do some of 
these variables appear to distinguish one breed from another? 

(b) View the data in three dimensions using the variables Breed, Frame, and BkFat. Ro¬ 
tate the coordinate axes in various directions. Check for outliers. Are the breeds well 
separated in this coordinate system? 

(c) Repeat part b using Breed, FtFrBody, and SaleHt. Which three-dimensional display 
appears to result in the best separation of the three breeds of bulls? 


Table 1 

.10 Data on Bulls 







Breed 

SalePr 

YrHgt 

FtFrBody 

PrctFFB 

Frame 

BkFat 

SaleHt 

SaleWt 

1 

2200 

51.0 

1128 

70.9 

7 

.25 

54.8 

1720 

1 

2250 

51.9 

1108 

72.1 

7 

.25 

55.3 

1575 

1 

1625 

49.9 

1011 

71.6 

6 

.15 

53.1 

1410 

1 

4600 

53.1 

993 

68.9 

8 

.35 

56.4 

1595 

1 

2150 

51.2 

996 

68.6 

7 

.25 

55.0 

1488 

8 

1450 

51.4 

997 

73.4 

7 

.10 

55.2 

1454 

8 

1200 

49.8 

991 

70.8 

6 

.15 

54.6 

1475 

8 

1425 

50.0 

928 

70.8 

6 

.10 

53.9 

1375 

8 

1250 

50,1 

990 

71.0 

6 

.10 

54.9 

1564 

8 

1500 

51.7 

992 

70.6 

7 

.15 

55.1 

1458 


Source: Data courtesy of Mark Ellersieck. 


1.27. Table 1.11 presents the 2005 attendance (millions) at the fifteen most visited national 
parks and their size (acres). 

(a) Create a scatter plot and calculate the correlation coefficient. 
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(b) Identify the park that is unusual. Drop this point and recalculate the correlation 
coefficient. Comment on the effect of this one point on correlation. 

(c) Would the correlation in Part b change if you measure size in square miles instead of 
acres? Explain. 


Table l.l I Attendance and Size of National Parks 


National Park 

Size (acres) 

Visitors (millions) 

Arcadia 

47.4 

2.05 

Bruce Canyon 

35.8 

1.02 

Cuyahoga Valley 

32.9 

2.53 

Everglades 

1508.5 

1.23 

Grand Canyon 

1217.4 

4.40 

Grand Teton 

310.0 

2.46 

Great Smoky 

521.8 

9.19 

Hot Springs 

5.6 

1.34 

Olympic 

922.7 

3.14 

Mount Rainier 

235.6 

1.17 

Rocky Mountain 

265.8 

2.80 

Shenandoah 

199.0 

1.09 

Yellowstone 

2219.8 

2.84 

Yosemite 

761.3 

3.30 

Zion 

146.6 

2.59 
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Matrix Algebra 
and Random Vectors 


2.1 Introduction 


We saw in Chapter 1 that multivariate data can be conveniently displayed as an 
array of numbers. In general, a rectangular array of numbers with, for instance, n 
rows and p columns is called a matrix of dimension n X p. The study of multivariate 
methods is greatly facilitated by the use of matrix algebra. 

The matrix algebra results presented in this chapter will enable us to concisely 
state statistical models. Moreover, the formal relations expressed in matrix terms 
are easily programmed on computers to allow the routine calculation of important 
statistical quantities. 

We begin by introducing some very basic concepts that are essential to both our 
geometrical interpretations and algebraic explanations of subsequent statistical 
techniques. If you have not been previously exposed to the rudiments of matrix al¬ 
gebra, you may prefer to follow the brief refresher in the next section by the more 
detailed review provided in Supplement 2A. 


2.2 Some Basics of Matrix and Vector Algebra 
Vectors 


An array x of n real numbers Xi , x 2 ,..,, x n is called a vector, and it is written as 

M 


X = 


*2 


or X' = [*,,x 2 ,•••,■*»] 


L*nJ 

where the prime denotes the operation of transposing a column to a row. 
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A vector x can be represented geometrically as a directed line in n dimensions 
with component x Y along the first axis, x 2 along the second axis,,and x„ along the 
rtth axis.This is illustrated in Figure 2.1 for n = 3. 

A vector can be expanded or contracted by multiplying it by a constant c. In 
particular, we define the vector cx as 



That is, cx is the vector obtained by multiplying each element of x by c. [See 
Figure 2.2(a).] 


2 



(a) (b) 


Figure 2.2 Scalar multiplication and vector addition. 
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Two vectors may be added. Addition of x and y is defined as 

*i + yi 
*2 + yi 

x n + yn 

so that x + y is the vector with ith element x t + y h 

The sum of two vectors emanating from the origin is the diagonal of the paral¬ 
lelogram formed with the two original vectors as adjacent sides. This geometrical 
interpretation is illustrated in Figure 2.2(b). 

A vector has both direction and length. In n = 2 dimensions, we consider the 
vector 




The length of x, written L x , is defined to be 

L x = Vjcj + x 2 

Geometrically, the length of a vector in two dimensions can be viewed as the 
hypotenuse of a right triangle. This is demonstrated schematically in Figure 2.3. 

The length of a vector x' = [*i, x 2 , ..., *„], with n components, is defined by 

L x = V4 + 4+ ■•■+ (2-D 

Multiplication of a vector x by a scalar c changes the length. From Equation (2-1), 

L cx = \/c 2 x\ + c 2 x 2 + * ■ • + c 2 x 2 n 

= |c|V*? + xj + ■■■ + x 2 n = |c|L, 


Multiplication by c does not change the direction of the vector x if c > 0. 
However, a negative value of c creates a vector with a direction opposite that of x. 
From 


= |c|L, (2-2) 

it is clear that x is expanded if |c| > 1 and contracted if 0 < \c\ < 1. [Recall 
Figure 2.2(a).] Choosing c = L ~ l , we obtain the unit vector L^x, which has length 1 
and lies in the direction of x. 
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cos (0) = 





Figure 2.4 The angle 6 between 
x' — [x lt x 2 ] and y' = [y*,yi]« 


A second geometrical concept is angle. Consider two vectors in a plane and the 
£ e between them, as in Figure 2.4. From the figure, 0 can be represented as 
^difference between the angles 0, and 0 2 formed by the two vectors and the first 
coordinate axis. Since, by definition, 

cos(0i) = y~ cos(0 2 ) = j- 


sin(^i) 


sin ( Q 2 ) = 


L 


cos(0) = cos(0 2 - 0i) = cos(0 2 ) cos(0i) + sin(0 2 ) sin(0i) 

. angle 0 between the two vectors x' = [*i, x 2 ] and y' = [yi, yi] is specified by 


cos(0) = cos (0 2 - 0i) 


/ il) + | ( — 1 = Xl3>1 - — (2-3) 

LJ \L X ) \ L y ) \L X ) L x L y 


We find it convenient to introduce the inner product of two vectors. For n - 2 
dimensions, the inner product of x and y is 

x'y = Xjy, + x 2 yz 

With this definition and Equation (2-3), 


L x L y Vx'x Vy y 

Since cos(90°) = cos (270°) = 0 and cos(0) = 0 only if x’y = 0, x and y are 
„pmendicular when x'y = 0. 

P For an arbitrary number of dimensions n, we define the inner product of x 
and y as 

x'y = -^1^1 + x 2Y2 + **• + x ny n 


inner product is denoted by either x'y or y'x. 
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Using the inner product, we have the natural extension of length and angle to 
vectors of n components: 


L x = length of x = Vx^x 

(2-5) 

x'y x'y 

cos (0) = = - - . - 

L x L y Vx'x Vy'y 

(2-6) 


Since, again, cos(0) = 0 only if x'y = 0, we say that x and y are perpendicular 
when x'y = 0. 


Example 2.1 (Calculating lengths of vectors and the angle between them) Given the 
vectors x' = [1,3,2] and y' = [-2,1, -1], find 3x and x + y. Next, determine 
the length of x, the length of y, and the angle between x and y. Also, check that 
the length of 3x is three times the length of x. 

First, 



Next, x'x = l 2 + 3 2 + 2 2 = 14, y'y = (-2) 2 + l 2 + (-1) 2 = 6, and x'y = 
1(—2) + 3(1) + 2(-l) = -1. Therefore, 

L x = = Vl4 = 3.742 L v = VyW = V6 = 2.449 


C0S ^ L x L y 3.742 X 2.449 ' 109 


so 9 = 96.3°. Finally, 

L 3x = V3 2 + 9 2 + 6 2 = V126 and 3 L x = 3V14 = Vl26 
showing L 2x = 3 L x . 


A pair of vectors x and y of the same dimension is said to be linearly dependent 
if there exist constants c x and c 2 , both not zero, such that 

c\ x + c 2 y = 0 

A set of vectors x 1} x 2 ,..., x* is said to be linearly dependent if there exist constants 
C\y c 2 ,..., Cjt, not all zero, such that 

c i x i + c 2 x 2 + • • • + c k Xk = 0 (2-7) 

Linear dependence implies that at least one vector in the set can be written as a 
linear combination of the other vectors. Vectors of the same dimension that are not 
linearly dependent are said to be linearly independent. 
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Example 2.2 (Identifying linearly independent vectors) Consider the set of vectors 


Setting 
implies that 


Xj = 2 x 2 ~ \ 0 I x 3 - -2 

_1J L-Ij L 1. 

CjX! + c 2 x 2 + C3X3 = 0 


C 1 + c 2 + C 3 — 0 
2cj - 2 c 3 = 0 
ci — c 2 + c 3 = 0 

with the unique solution c { = c 2 = c 3 = 0. As we cannot find three constants c 1 , c 2 , 
and c 3 , not all zero ,such that C\ x x + c 2 x 2 + c 3 x 3 = 0, the vectors x ly x 2 , and x 3 are 
lin early in dep en den t. ■ 

The projection (or shadow) of a vector x on a vector y is 

(x'y) (x'y) 1 

Projection of x on y = — 7 — y - — ~ y~ y 

y y L y 

where the vector Ly l y has unit length. The length of the projection is 


I x'y I r x y 

Length of projection = —-— - L x —r 

L,y 

where 6 is the angle between x and y. (See Figure 2.5.) 


L x | cos(#)| 



- L* cos ( 6 ) - 

Matrices 


Figure 2.5 The projection of x on y. 


A matrix is any rectangular array of real numbers. We denote an arbitrary array of n 
rows and p columns by 


*11 

*12 

*• 

*21 

*22 •• 

’* a 2p 

.*/il 

a n2 * 

a np. 
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Many of the vector concepts just introduced have direct generalizations to matrices. 

The transpose operation A' of a matrix changes the columns into rows, so that 
the first column of A becomes the first row of A', the second column becomes the 
second row, and so forth. 



A matrix may also be multiplied by a constant c. The product cA is the matrix 
that results from multiplying each element of A by c.Thus 

can ca 12 ca lp 

_ ca 21 ca 22 *■* ca 2p 

(nx P ) : : : 

_c< 2 ni ca n2 ■** ctt n p_ 

Two matrices A and B of the same dimensions can be added. The sum A + B has 
(z\/)th entry a tj + b i} . 


Example 2.4 (The sum of two matrices and multiplication of a matrix by a constant) 

If 


A = 

'0 

3 

r 

and B = 

"l 

-2 

-3" 

(2x3) 

1 

-1 

i_ 

(2X3) 

2 

5 

1 _ 


then 

4A=[; « <1 and 

(2x3) 14 —4 4 

a + b =” 0 + l 3 - 2 i- 3 ~ ri 1 -2 ~l 

(2x3) (2X3) _1 + 2 -1 + 5 1 + 1 _3 4 2J 

It is also possible to define the multiplication of two matrices if the dimensions 
of the matrices conform in the following manner: When A is (n X k) and B is 
(k X p) y so that the number of elements in a row of A is the same as the number of 
elements in a column of B, we can form the matrix product AB. An element of the 
new matrix AB is formed by taking the inner product of each row of A with each 
column of B. 
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The matrix product AB is 


A B = the (n x p) matrix whose entry in the ith row 
(nxk)(kxp) anc j yth column is the inner product of the ith row 
of A and the ;th column of B 


(/,;') entry of AB = a n b Xj + a i2 b 2j + + a ik b kj = 2 a tt b tj ( 2 ’ 10 ) 

When k = 4, we have four products to add for each entry in the matrix AB. Thus, 


a \\ a U a \3 a \4 r, /T\ h 

: : bn ’■* b\j *“ °ip 

- ' ■ b 2 j •** 

A B = ^2 ^3 5/3) , - h 

(#lX4)(4Xp) V-•-■-; 0 31 P 3 y 0 3p 




b A i *’* [&4/J *’* ^4/ 


Column 


= Row/ + a nhj + fl f3^3; + fl/4^4/)* 


Example 2.5 (Matrix multiplication) If 


3-12 


1 5 4’ 


7 L and C = 


2 0 


3-12 
A B = i . . 

(2x3)(3xl) [1 5 4 


3( 2) + ( 1) (7) + 2(9) 
l(-2)+5(7) +4(9) 


2 0 3 -1 2 

C A = 

(2x2)(2x3) 1-1 1 5 4 


2(3) +0(1) 2(-l) + 0(5) 2(2) + 0(4) 
1(3) -1(1) 1(-1) - 1(5) 1(2) -1(4) 

6 -2 4 ] 

2 -6 -2 J 

(2X3) 
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When a matrix B consists of a single column, it is customary to use the lower¬ 
case b vector notation. 


Example 2.6 (Some typical products and their dimensions) Let 


1 -2 3 

2 4-1 


b = -3 c = 8 

6 -4 


-B] 


Then Ab, be', b'c, and d'A b are typical products. 


1 -2 3 

2 4-1 




The product A b is a vector with dimension equal to the number of rows of A. 

r si 


b'c =[7-3 6] 8 = [-13] 

-4 


The product b'c is a 1 X 1 vector or a single number, here -13. 


35 

56 

-28 

-15 

-24 

12 

30 

48 

-24 


The product b c' is a matrix whose row dimension equals the dimension of b and 
whose column dimension equals that of c. This product is unlike b'c, which is a 
single number. 


d'A b = [2 9] 


*1 -2 3~1 7 

|_2 4 -ij =f 26 J 


The product d'A b is a 1 x 1 vector or a single number, here 26. 


Square matrices will be of special importance in our development of statistical 
methods. A square matrix is said to be symmetric if A = A' or = a }i for all i 
and j. 
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Example 2.7 (A symmetric matrix) The matrix 



is symmetric; the matrix 



is not symmetric. 


When two square matrices A and B are of the same dimension, both products 
AB and BA are defined, although they need not be equal. (See Supplement 2A.) 
If we let I denote the square matrix with ones on the diagonal and zeros elsewhere, 
it follows from the definition of matrix multiplication that the (z, ;)th entry of 
AI is a n X 0 + ■ • ■ + a iy j-i X 0 + a i} - X 1 + q ;+1 X 0 + • • • + a ik X 0 = q ; , so 
AI = A. Similarly, IA = A, so 

I A = A I = A for any A (2-11) 

(*x*)(*x*) (kxk)(kxk) {kxk) (kxk) 

The matrix I acts like 1 in ordinary multiplication (1 - a = a -1 = a), so it is 
called the identity matrix. 

The fundamental scalar relation about the existence of an inverse number a' 1 
such that aT l a = aaT 1 = 1 if a ^ 0 has the following matrix algebra extension: If 
there exists a matrix B such that 

B A = A B = I 

(*x*)(*x*) (kxk){kxk) (kxk) 

then B is called the inverse of A and is denoted by A -1 . 

The technical condition that an inverse exists is that the k columns a 2 , a 2 ,..., a* 
of A are linearly independent. That is, the existence of A -1 is equivalent to 



qa : + c 2 a 2 + • + c k a* = 0 only if q = = c k = 0 (2-12) 


(See Result 2A.9 in Supplement 2A.) 


Example 2.8 (The existence of a matrix inverse) For 



you may verify that 

’-.2 . 4*1 ("3 2l r (-.2)3 + (.4)4 (-.2)2 + (.4)1 

.8 -.6J [4 lj [ (* 8 ) 3 + (-- 6 ) 4 (* 8 ) 2 + (~ 6 )!_ 
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is A 1 . We note that 


-.2 .4 

.8 -.6 


Cl 4 +C2 1 = 0 


implies that q == c 2 = 0, so the columns of A are linearly independent. This 
confirms the condition stated in (2-12). ■ 

A method for computing an inverse, when one exists, is given in Supplement 2A. 
The routine, but lengthy, calculations are usually relegated to a computer, especially 
when the dimension is greater than three. Even so, you must be forewarned that if 
the column sum in (2-12) is nearly 0 for some constants q,..., c k , then the computer 
may produce incorrect inverses due to extreme errors in rounding. It is always good 
to check the products AA~ ! and A -1 A for equality with I when A -1 is produced by a 
computer package. (See Exercise 2.10.) 

Diagonal matrices have inverses that are easy to compute. For example. 


033 0 

0 a 44 


has inverse 


if all the an ^ 0. 

Another special class of square matrices with which we shall become familiar 
are the orthogonal matrices, characterized by 

QQ' = QQ = I or Q' = Q" 1 (2-13) 


The name derives from the property that if Q has zth row q/, then QQ' = I implies 
that qjq, - 1 and q[q ; == 0 for i ^ so the rows have unit length and are mutually 
perpendicular (orthogonal). According to the condition Q'Q = I, the columns have 
the same property. 

We conclude our brief introduction to the elements of matrix algebra by intro¬ 
ducing a concept fundamental to multivariate statistical analysis. A square matrix A 
is said to have an eigenvalue A, with corresponding eigenvector x ^ 0, if 


Ax = Ax 


(2-14) 
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Ordinarily, we normalize x so that it has length unity; that is, 1 = x'x. It is 
convenient to denote normalized eigenvectors by e, and we do so in what follows. 
Sparing you the details of the derivation (see [1]), we state the following basic result: 

Let A be a A: X A: square symmetric matrix. Then A has A pairs of eigenvalues 
and eigenvectors namely, 

Ai,e, A 2 ,e 2 ... A*,e t (2-15) 

The eigenvectors can be chosen to satisfy 1 = eje! = = e^e* and be mutually 

perpendicular. The eigenvectors are unique unless two or more eigenvalues 
are equal. 


Example 2.9 (Verifying eigenvalues and eigenvectors) Let 



Then, since 



L V5J L V5J 


Ai = 6 is an eigenvalue, and 



is its corresponding normalized eigenvector. You may wish to show that a second 
eigenvalue-eigenvector pair is A 2 = -4, e 2 = [1/V5,1/V2]. ■ 


A method for calculating the A’s and e’s is described in Supplement 2A. It is in¬ 
structive to do a few sample calculations to understand the technique. We usually rely 
on a computer when the dimension of the square matrix is greater than two or three. 


.3 Positive Definite Matrices 

The study of the variation and interrelationships in multivariate data is often based 
upon distances and the assumption that the data are multivariate normally distributed. 
Squared distances (see Chapter 1) and the multivariate normal density can be 
expressed in terms of matrix products called quadratic forms (see Chapter 4). 
Consequently, it should not be surprising that quadratic forms play a central role in 
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multivariate analysis. In this section, we consider quadratic forms that are always 
nonnegative and the associated positive definite matrices. 

Results involving quadratic forms and symmetric matrices are, in many cases, 
a direct consequence of an expansion for symmetric matrices known as the 
spectral decomposition . The spectral decomposition of a A: X A: symmetric matrix 
A is given by 1 

A = Aj ei ej + A 2 e 2 ®2 + * * * + A^ (2-16) 

(*x*) (*X1)(1X*) (*X1)(1X*) (*X1)(1X*) 


where A l5 A 2 ,..., A* are the eigenvalues of A and e!, e 2l ..., e* are the associated 
normalized eigenvectors. (See also Result 2A.14 in Supplement 2A). Thus, e-e,- = 1 
for i = 1,2,..., A, and e[e y = 0 for i =£ j. 


Example 2.10 (The spectral decomposition of a matrix) Consider the symmetric matrix 

" 13 -4 2~ 

A = -4 13 -2 

2 -2 10 _ 

The eigenvalues obtained from the characteristic equation | A — AI | = 0 are 
Ai = 9, A 2 = 9, and A 3 = 18 (Definition 2A.30). The corresponding eigenvectors 
e!,e 2 , and e 3 are the (normalized) solutions of the equations Ae, = A/e* for 
i = 1,2, 3. Thus, Aex = Ae x gives 


13 ■ 4 2 €n Cj i 

4 13 ~2 e 2 i — 9 <?2i 

2 ~-2 10 _ _ c 3 j _ _ £ 3 1 


13e n ~ 4e 2 i + 2e 31 - 9e n 
~4e n + 13e 2 i ~ 2 * 3 i = 9*21 
2eu - 2*21 + 10e 31 = 9e 31 

Moving the terms on the right of the equals sign to the left yields three homogeneous 
equations in three unknowns, but two of the equations are redundant. Selecting one of 
the equations and arbitrarily setting * n = 1 and e 21 — 1, we find that * 31 = 0. Con¬ 
sequently, the normalized eigenvector is ej = [l/\/l 2 + l 2 + 0 2 , l/\/l 2 + l 2 + 0 2 , 
0/Vl 2 + l 2 + 0 2 ] — [1/V5,1/V2,0], since the sum of the squares of its elements 
is unity. You may verify that e 2 = [1/ Vl8, -1/V18, -4/ Vl8] is also an eigenvector 
for 9 = A 2 , and e 3 = [2/3, —2/3, 1/3] is the normalized eigenvector corresponding 
to the eigenvalue A 3 = 18. Moreover, eje ; - = 0 for i ¥= j. 


1 A proof of Equation (2-16) is beyond the scope of this book. The interested reader will find a proof 
in [6], Chapter 8. 
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The spectral decomposition of A is then 

A = Ajejei + A 2 e 2 e2 + A 3 e 3 e3 



as you may readily verify. 


The spectral decomposition is an important analytical tool. With it, we are very 
easily able to demonstrate certain statistical results. The first of these is a matrix 
explanation of distance, which we now develop. 

Because x'Ax has only squared terms xf and product terms x t x k , it is called a 
quadratic form. When a k X k symmetric matrix A is such that 

0 < x'Ax (2-17) 

for all x' = [*!, x 2 , ■ . •, **], both the matrix A and the quadratic form are said to be 
nonnegative definite . If equality holds in (2-17) only for the vector x' = [0,0,..., 0], 
then A or the quadratic form is said to be positive definite. In other words, A is 
positive definite if 

0 < x'Ax (2-18) 


for all vectors x ^ 0. 
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Example 2.11 (A positive definite matrix and quadratic form) Show that the matrix 
for the following quadratic form is positive definite: 

3x] + 2x 2 2 - 2V2 Xl x 2 

To illustrate the general approach, we first write the quadratic form in matrix 
notation as 


[*i *2] 




x'Ax 


By Definition 2A.30, the eigenvalues of A are the solutions of the equation 
| A — Al | = 0, or (3 — A) (2 — A) — 2 = 0. The solutions are Aj = 4 and A 2 “ 1. 
Using the spectral decomposition in (2-16), we can write 

A — Ajej ej + A 2 e 2 e 2 

(2X2) (2xl)(lx2) (2xl)(lx2) 


= 4e 1 ej + e 2 e£ 

(2X1)(1X2) (2xl)(lx2) 

where e : and e 2 are the normalized and orthogonal eigenvectors associated with the 
eigenvalues = 4 and A 2 = 1, respectively. Because 4 and 1 are scalars, premulti¬ 
plication and postmultiplication of A by x' and x, respectively, where x' = [*1 > * 2 ] * s 
any nonzero vector, give 

x' A X = 4x' e, el I + x' e 2 e 2 x 

(Ix2)(2x2)(2xl) (1X2)(2X1)(1X2)(2X1) (1X2)(2X1)(1X2)(2X1) 

= 4y? + y\ a 0 

with 


yi = x , e 1 = ejx and = x'e 2 = e 2 x 


We now show that y x and are not both zero and, consequently, that 
x'Ax = 4y\ + y\ > 0, or A is positive definite. 

From the definitions of yi and , we have 


or 



y = E x 

(2X1) (2X2)(2X1) 


Now E is an orthogonal matrix and hence has inverse E\ Thus, x = E'y. But x is a 
nonzero vector, and 0 ^ x = E'y implies that y ^ 0. ■ 


Using the spectral decomposition, we can easily show that a k X k symmetric 
matrix A is a positive definite matrix if and only if every eigenvalue of A is positive. 
(See Exercise 2.17.) A is a nonnegative definite matrix if and only if all of its eigen¬ 
values are greater than or equal to zero. 

Assume for the moment that the p elements x\ t x 2 , ■ • -, x p of a vector x are 
realizations of p random variables Xy, X 2i ..., X p . As we pointed out in Chapter 1, 
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we can regard these elements as the coordinates of a point in p-dimensional space, 
and the "distance” of the point [x x , * 2 , ■ ■ • * X P \ to the ori g in can > and in this ca se 
should, be interpreted in terms of standard deviation units. In this way, we can 
account for the inherent uncertainty (variability) in the observations. Points with the 
same associated "uncertainty” are regarded as being at the same distance from 
the origin. 

If we use the distance formula introduced in Chapter 1 [see Equation (1-22)], 
the distance from the origin satisfies the general formula 

(distance) 2 = a n x\ + 022*2 + '** + a pp x p 

+ 2(012*1*2 + 013*1*3 + + 0p-l,p*p-l*p) 


provided that (distance) 2 > Oforall [*i, * 2 ,...,* p ] * [0,0,..., 0]. Setting^ 
i * i = 1 , 2 ,...,/?, j = 1,2,we have 




or 

0 < (distance) 2 = x'Ax forx ^ 0 (2-19) 

From (2-19), we see that the p X p symmetric matrix A is positive definite. In 
sum, distance is determined from a positive definite quadratic form x'Ax. Con¬ 
versely, a positive definite quadratic form can be interpreted as a squared distance. 

Comment. Let the square of the distance from the point x' = [x x , x 2 ,..., * p ] 
to the origin be given by x'Ax, where A is a p X p symmetric positive definite 
matrix. Then the square of the distance from x to an arbitrary fixed point 
p' = [pi,p 2 ,...,p p ] is given by the general expression (x - p)'A(x - p). 

Expressing distance as the square root of a positive definite quadratic form al¬ 
lows us to give a geometrical interpretation based on the eigenvalues and eigenvec¬ 
tors of the matrix A. For example, suppose p = 2. Then the points x' = [x 1 , x 2 ] of 
constant distance c from the origin satisfy 

x'Ax = 011*5 + 0 22 *2 + 2012*1*2 = C 2 

By the spectral decomposition, as in Example 2.11, 

A = A iei eJ + A 2 e 2 e 2 so x'Ax = A^x'eO 2 + A 2 (x'e 2 ) 2 

Now, c 2 = A x y5 + A 2 y 2 is an elli P se in A = X Cl and & = x ' e 2 because Ai, A 2 > 0 
when A is positive definite. (See Exercise 2.17.) We easily verify that x = cAr 1/2 ei 
satisfies x'Ax = = c 2 . Similarly^ = cX 2 l/2 e 2 gives the appropriate 

distance in the e 2 direction. Thus, the points at distance c lie on an ellipse whose axes 
are given by the eigenvectors of A with lengths proportional to the reciprocals of 
the square roots of the eigenvalues. The constant of proportionality is c. The situa¬ 
tion is illustrated in Figure 2.6. 
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Figure 2.6 Points a 
constant distance c 
from the origin 
(p = 2, 1 < Ai < A 2 ) ■ 



If p > 2, the points x' = [*i, * 2 , - ■., * p ] a constant distance c ~ Vx'Ax from 

9^2 2 

the origin lie on hyperellipsoids c = A^x'ei) + ■*• + A p (x'e p ) , whose axes are 
given by the eigenvectors of A.The half-length in the direction e £ - is equal to c/ Va], 
i = 1, 2,..., p, where \\, A 2 ,..., A p are the eigenvalues of A. 

2.4 A Square-Root Matrix 

The spectral decomposition allows us to express the inverse of a square matrix in 
terms of its eigenvalues and eigenvectors, and this leads to a useful square-root 
matrix. 

Let A be a A: X k positive definite matrix with the spectral decomposition 
k 

A = 2 A/e,e■. Let the normalized eigenvectors be the columns of another matrix 
i=l 

P = [e 1 ,e 2) ...,e*].Then 

A = 2 A, e ( - e' = P A P' (2-20) 

(kxk) fTi (*xl)(lx*) (kxk)(kxk)(kxk) 

where PP' = P'P = I and A is the diagonal matrix 

"Aj 0 • • - 0 

° A 2 * * • ° 

(. kxk) : : \ : 

.0 0 A* 


with A i > 0 
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Thus, 

A' 1 = PA -1 P' = 2 7 e < e < ! ( 2_21 ) 

1=1 A l 

since (PA-'P')PAP' = PAP'CPA 'P ) = PP’ = I. 

Next, let A 1 / 2 denote the diagonal matrix with Va" as the ith diagonal element. 

TTie matrix 2 VAj e,-ei = PA 1/2 P' is caUed the square root of A and is denoted by 

i=i 

A 1/2 . 


The square-root matrix, of a positive definite matrix A, 

A 1 ' 2 = 2 Va; e,ej = PA^P' (2-22) 

;=i 


has the following properties: 

1 . (A 1/2 )' = A 1/2 (that is, A 1/2 is symmetric). 

2. A^A 1 ^ = A. 


3. (A 1/2 ) -1 = 2 - 4 = e,-e' = PA _1/2 P', where A _V2 is a diagonal matrix with 

i=l vAj 

1 /VA^ as the ith diagonal element. 

4. a 1 /2 A“ 1/2 = A' 1/2 A 1/2 = I, and A" 1/2 A" 1/2 = A -1 , where A“ 1/2 = (A 1/2 ) \ 


2.5 Random Vectors and Matrices 

A random vector is a vector whose elements are random variables. Similarly, a 
random matrix is a matrix whose elements are random variables. The expected value 
of a random matrix (or vector) is the matrix (vector) consisting of the expected 
values of each of its elements. Specifically, let X = {.XyJ.be an n X p random 
matrix. Then the expected value of X, denoted by £(X), is the n X p matrix of 
numbers (if they exist) 

'E(X n ) E(X n ) ■■■ E(X lp )~ 

E(X 21 ) E(X 22 ) E(X2 P ) 

_E(Xn 1 ) E(X„ 2 ) ■■■ E(X np )_ 


£(X) = 


(2-23) 
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where, for each element of the matrix, 2 


E(Xu) 


/ CO 

x . .f. { x . .)dx r if %ij is a continuous random variable with 
00 11 t} l} t} probability density function fij(Xjj) 

if Xij is a discrete random variable with 
2j x ijPij( x ij) probability function 


Example 2.12 (Computing expected values for discrete random variables) Suppose 
p = 2 and n = 1, and consider the random vector X' = [Xi , X 2 ). Let the discrete 
random variable Xi have the following probability function: 


™en E(X,) = £ = (-1)(.3) + (0)(.3) + (1)(.4) = .1. 

alUj 

Similarly, let the discrete random variable X 2 have the probability function 


x 2 

0 

1 

P2(X 2 ) 

.8 

.2 


Then E{X 2 ) = 2 * 2 ft(* 2 ) = (0)(.8) + (1)(.2) = .2. 

all X'l 


E(X i) = .1 

E(x 2 )\ [.2 


TWo results involving the expectation of sums and products of matrices follow 
directly from the definition of the expected value of a random matrix and the univariate 
properties of expectation, E(X 1 + Y x ) = E(Xi) + E(Yi) and E(cX i) = cE(Xi). 
Let X and Y be random matrices of the same dimension, and let A and B be 
conformable matrices of constants. Then (see Exercise 2.40) 

E(X + Y) = E(X) + E( Y) (2-24) 

E(AXB) = A£(X)B 


2 If you are unfamiliar with calculus, you should concentrate on the interpretation of the expected 
value and, eventually, variance. Our development is based primarily on the properties of expectation 
rather than its particular evaluation for continuous or discrete random variables. 
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2.6 Mean Vectors and Covariance Matrices 

Suppose X' = [ AT, X 2i ..., X p ] is a p X 1 random vector. Then each element of X is a 
random variable with its own marginal probability distribution. (See Example 2.12.) The 
marginal means and variances a\ are defined as - E(X,) andaf — E(Xi - p,*) 2 , 
i = 1,2,..p, respectively. Specifically, 

r pOO 

I x t fi( Xi) dxi Xi is a continuous random variable with probability 
oo 1 1 density function//(x,) 

Pi = 

if Xi is a discrete random variable with probability 
Zj x iPi( x i) function #■(*;) 

V all Xj 

U °° (2-25) 

( x . _ ^.)2 f.( Xi ) dxi is a continuous random variable 

oo with probability density function 


2 (*< - Vi) 2 Pi( x i) 

< all 


if Xi is a discrete random variable 
with probability function Pi{x ( ) 


It will be convenient in later sections to denote the marginal variances by cr,, rather 
than the more traditional oj , and consequently, we shall adopt this notation. 

The behavior of any pair of random variables, such as AT and AT, is described by 
their joint probability function, and a measure of the linear association between 
them is provided by the covariance 

o-ik = E(X t - fj.i)(X k - fi k ) 

r » OO f oo 

/ / (j X t - fti)(x k - n k )f ik (Xj,x k )dXidx k if X it X k are continuous 

J~co oo random variables with 

the joint density 

_ > function f ik (xi,x k ) 


2 2 (*. - Mi) (** - Pk)Pik(xi, X k ) if X t , X k are discrete 

ail ail x k random variables with 

joint probability 

, function p lk (x iy x k ) 

(2-26) 

and ^ and p k , i,k = 1,2,..., p, are the marginal means. When i = k, the covari¬ 
ance becomes the marginal variance. 

More generally, the collective behavior of the p random variables Xi , X 2i .>.,X p 
or, equivalently, the random vector X' = [AT, X 2 ,..., X p \, is described by a joint 
probability density function f(x 1 , x 2y ..., x p ) = /(x). As we have already noted in 
this book,/(x) will often be the multivariate normal density function. (See Chapter 4.) 

If the joint probability P[ AT < x,-and AT ^ x k ] can be written as the product of 
the corresponding marginal probabilities, so that 

P[x, £ X, and X k £ a:*] = P[X t £ x,}P[X k £ a:*] 


(2-27) 
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for all pairs of values x n x k , then AT and X k are said to be statistically independent. 
When X t and X k are continuous random variables with joint density /•*(*, , x k ) and 
marginal densities /;(x,) and f k {x k ) y the independence condition becomes 

fik( x i » x k) ~~ fi{ x i)fk{ x k) 

for all pairs (x iy x k ). 

The p continuous random variables AT, X 2 , ...,X p are mutually statistically 
independent if their joint density can be factored as 

fn - p(xux 2 ,...,x p ) = fi(x l )f 2 (x 2 ) ■■■ fp(x p ) (2-28) 

for all p-tuples (x ll x 2 ,...,x p ). 

Statistical independence has an important implication for covariance. The 
factorization in (2-28) implies that Cov (AT, X k ) ~ O.Thus, 

Cov (AT, X k ) = 0 if X t and X k are independent (2-29) 


The converse of (2-29) is not true in general; there are situations where 
Cov (AT, X k ) = 0, but Xi and X k are not independent. (See [5].) 

The means and covariances of the p X 1 random vector X can be set out as 
matrices. The expected value of each element is contained in the vector of means 
P = £(X), and the p variances era and the p(p - l)/2 distinct covariances 
<T ik {i < k) are contained in the symmetric variance-covariance matrix 
X = £(X - p)(X - p)'. Specifically, 



S = £(X-|0(X-|0' 

/ Xi - pi 

= £ [ X 1 -ix 2 [Jf . ^ x p -» p ] 

1 Xp - Mp_ 

(Xi — fii) 2 (Xi ~ ni)(X 2 — nz) ■■■ (Xi — fii)(X p — fx p ) 

(X 2 - p. 2 ){X x — Mi) (X 2 — M2) 2 ■■■ (X 2 — fi 2 )(X p — /x p ) 

(Xp-v p )(X1 -Mi) (X p -»p)(X 2 -^) ••• ( Xp-tXp ) 2 _ 

E(X j - Mi) 2 E(X \ - Mi) (X 2 - M2) • • • E(X \ - fii)(X p - m p ) 

E(X 2 - ^)(Xi - Ml) E(X 2 M 2) 2 E(x 2 -H2)(X P - ftp) 

E{X p - m p )(A'i - Mi) E(X p - fx p )(X 2 - M2) --- E(Xp-fip ) 2 
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0*11 

a 12 *' 

* "Ip 

X = Cov(X) = 

<^21 

^22 ’* 

p 


-°>i 


a PP- 


Example 2.13 (Computing the covariance matrix) Find the covariance matrix for 
the two random variables X\ and X 2 introduced in Example 2.12 when their joint 
probability function /? 12 (*i, * 2 ) represented by the entries in the body of the 
following table: 


\ *2 

x \ 

0 

1 

P\( x \) 

-1 

.24 

.06 

3 

0 

.16 

.14 

.3 

1 

.40 

.00 

.4 

Pl( X 2) 

.8 

.2 

1 


We have already shown that p x = E{X{) = .1 and i± 2 = E(X 2 ) = .2. (See Exam¬ 
ple 2.12.) In addition. 


o-n = E(X, - mi) 2 = 2 (*i - 

all x ] 

= (-1 - ,1) 2 (.3) + (0 - l) 2 (-3) + (1 - .1) 2 (.4) = .69 
cr 22 ~ E(X 2 — M 2) 2 = 2 C*2 — -2) 2 M 2 (*2) 

all *2 

= (0 - -2) 2 (.8) + (1 - -2) 2 (.2) 

= .16 

<7i2 = £(^1 — Ml) (^2 ~ M 2 ) = 2 C*1 ” *l)(-*2 “ '2 )/?i 2 (x 1 , JC 2 ) 

aj) pairs (JTi, x 2 ) 

= (-1 - .1) (0 - .2) (.24) + (-1 - .1)(1 - .2) (.06) 

4- ••• + (1 - .1)(1 - .2)(.00) = -.08 
0*21 = E(X 2 - p 2 )(X\ — pi) = E{X y — p{)(X 2 — p 2 ) = cr 12 = —.08 



Consequently, with X' = Jf 2 ]» 


and 


M = £(X) 


~E{X 1 )' 


Mi 


".1 

_E(X 2 )_ 


_M2_ 


.2_ 


S = £(X-m)(X-m)' 


= E 


(X, - Ml) 2 
(*2-M2)(*! - Ml) 


(*1 -M,)(^2-M2) 
(X 2 — M 2) 2 


£(*, - mi) 2 £(X, - Mi) (Xt - M2) 

E(X 2 - M 2 )(Xi - Mi) £(*2 -M 2) 2 

0*1 1 <7!2*1 _ .69 -.08 

cr 2 i cr 22 J .08 .16_ 


We note that the computation of means, variances, and covariances for discrete 
random variables involves summation (as in Examples 2.12 and 2.13), while analo¬ 
gous computations for continuous random variables involve integration. 

Because a ik = E(Xi — p t )(X k — p k ) = a kh it is convenient to write the 
matrix appearing in (2-31) as 


2 = E(X - m)(X - m)' = 

<711 

0*12 

<7j2 ‘ 

& 22 * - 

■ <n P 

‘ 

(2-32) 


-Pip 

*2 P ■■ 

a pp- 



We shall refer to p and X as the population mean (vector) and population 
variance-covariance (matrix), respectively. 

The multivariate normal distribution is completely specified once the mean 
vector p and variance-covariance matrix X are given (see Chapter 4), so it is not 
surprising that these quantities play an important role in many multivariate 
procedures. 

It is frequently informative to separate the information contained in vari¬ 
ances < 7 ,j from that contained in measures of association and, in particular, the 
measure of association known as the population correlation coefficient p ik . The 
correlation coefficient p ik is defined in terms of the covariance a ik and variances 
< 7/1 and cr kk as 


= °* ik 

Pik 


(2-33) 


The correlation coefficient measures the amount of linear association between the 
random variables X t and X k . (See, for example, [5].) 


n 
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Let the population correlation matrix be the p X p symmetric matrix 


cr\\ °~12 

Vo\i VoTi VcT22 

_ <T\l °~22 

p « VOn V^22 V0^2 V^22 

_ O-lp °~2p 

Vo^Vo^Tp 


1 Pi 2 Pip 

Pi2 1 ^2p 


LPlp P2p *■* 1 J 

and let the p X p standard deviation matrix be 


yl/2 = 


0 

0 Vcr 22 


L o o 

Then it is easily verified (see Exercise 223) that 

v V2pV 1/2 = 2 


fjP 


V^22 


B, 




p = (V 1 / 2 ) _ 1 Z(V 1/2 ) 1 ( 2 ‘ 37 ) 

That is, Z can be obtained from V 1/2 and p, whereas p can be obtained from Z. 
Moreover, the expression of these relationships in terms of matrix operations allows 
the calculations to be conveniently implemented on a computer. 


Example 2.14 (Computing the correlation matrix from the covariance matrix) 

Suppose 


1 

2~ 

r^n 

0*12 

°~13 

9 

-3 

- a 12 

0*22 

023 

-3 

25 _ 

L (T 13 

0*23 

0*33 


Obtain V 1 / 2 and p. 
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0 

0 

f 2 

0 

<f 

yfr 2 = o 


0 

= 0 

3 

0 

■ 4 . 0 

0 


|_o 

0 

5_ 


^00 

(yl/2)- 1 = 0 | 0 

-° 0 5 


Consequently, from (2-37), the correlation matrix p is given by 


( V i/2) ^(V 1 / 2 ) 1 = 0 I 0 1 


2 -3 25 I 10 0 


0 | 0 


Partitioning the Covariance Matrix 

Often, the characteristics measured on individual trials will fall naturally into two 
or more groups. As examples, consider measurements of variables representing 
consumption and income or variables representing personality traits and physical 
characteristics. One approach to handling these situations is to let the character¬ 
istics defining the distinct groups be subsets of the total collection of characteris¬ 
tics. If the total collection is represented by a (p X 1)-dimensional random 
vector X, the subsets can be regarded as components of X and can be sorted by 
partitioning X. 

In general, we can partition the p characteristics contained in the p X 1 random 
vector X into, for instance, two groups of size q and p — q> respectively. For exam¬ 
ple, we can write 


and a* = E(X) = 
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From the definitions of the transpose and matrix multiplication, 

(X<» - m (1) ) (X< 2 > - m (2) )' 

~Xi - Ml" 

^ ^ [X q+ i~— f^q+i, Xq+2 ~ Mg+ 2 » * • • ~ Mp] 

_X q ~ Mg_ 

(Xi — fll)(Xq+l ““ Mg + l) (^1 - Ml)(^C ?+2 ~ M 4 +. 2 ) *’■ (^1 “ Ml)(^p ~ Mp) 

(A " 2 - M 2 ) (^9 + 1 “ Mg + l) (%2 ~ M 2 ) (^+2 “ M 4 + 2 ) *** (X 2 ■“ M 2 )(^p ” Mp) 

_(Xg - M<j)(^+i - M^+i) (X g ~ fig)(X q+ 2~ M?+ 2 ) *” (^<7 “ M$)(^p Mp) 


Upon taking the expectation of the matrix (X (1) - m ( 1) ) (X (2) - M (2) )P we get 

^hq+l a hq+2 '** a lp 

£(X^-m ( 1, )(X (2) -M (2, )'= ‘V 1 <T2,r2 ’ 1 P = ^12 (2-39) 

-°Vg+l &q,q+2 *** ^gp- 

which gives all the covariances,^/, i = 1,2= 3 + 1,9 + 2,..., p, between 

a component of X (1) and a component of X (2) . Note that the matrix S 12 is not 
necessarily symmetric or even square. 

Making use of the partitioning in Equation (2-38), we can easily demonstrate that 
(X-m)(X-m)' 




(x (2) - p (2) ) (x (1) - u {1) y (x {2) - m ( 2 ) )(x ( 2) - M (2) y 

_ ((p-«)*l) (lx/ <(P-9)*1) (1X(P“9)) . 


and consequently, 


2 « £(X - M)(X - m)' = D I v 

pxp) p-nl 2. 



p-q 

^11 

i x 12 

X 2 i 

”1 £ 22 . 


0-11 

• * }g 

a l,q+l 



a qq 

a q.q + l 


°'q+ 1,1 

a q+lq 

^+ 1 , 4+1 

ct 9 +i,p 

. a pi 

a pq 

a p>4+l 

°>p 
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Note that X 12 = X 21 . The covariance matrix of X (1 * is Sn, that of X^ is X 22 , and 
that of elements from X^ and X^ 2 ^ is X 12 (or £ 21 V 

It is sometimes convenient to use the Cov (X'^X^) notation where 

Cov(X (1) ,X (2 >) = X 12 

is a matrix containing all of the covariances between a component of X* 1 * and a 
component of X^. 


The Mean Vector and Covariance Matrix 
for Linear Combinations of Random Variables 

Recall that if a single random variable, such as X\ , is multiplied by a constant c, then 

E{cX,) = cE(X x ) = c Ml 

and 

Var (cX Y ) = E(cX x - c^) 2 = c 2 Var(Z 1 ) = c 2 a n 

If X 2 is a second random variable and a and b are constants, then, using additional 
properties of expectation, we get 

Cov (aXi , bX 2 ) = E(aXi - a^)(bX 2 - bfi 2 ) 

= abE{X,-ni){X 2 - M2) 

= abCov (Xi, X 2 ) = abcr l2 

Finally, for the linear combination aXi + bX 2 , we have 

£(**! + MT 2 ) = fl£(Jiri) + bE(X 2 ) = fl/ij + b/i2 
Var(a^ + Z>JV 2 ) = E[( flAr 2 + - (a /n + b^)] 2 

= - mi) + ^(^2 - M2)] 2 

= £[* 2 (*i - Aii ) 2 + b\X 2 - M2) 2 + 2ai>(* 1 - Ml) (^2 - M2)] 
= c 2 Var(ATj) + fc 2 Var(A' 2 ) + 2afcCov(A r 1 , X 2 ) 

= a 2 crn + Z> 2 cr 22 + 2 abcr 12 ( 2 - 41 ) 

With c' = [a, h], aX y + bX 2 can be written as 


Similarly, E(aX 1 + bX 2 ) = a^i + bfx 2 can be expressed as 

[a b] Ml = c'm 

LM2. 

If we let 

x = P 711 an ~\ 
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be the variance-covariance matrix of X, Equation (2-41) becomes 

Var (aX 1 + bX 2 ) = Var(c'X) = c'Xc (2-42) 

since 

c'Xc = [a b] <7u ° 12 ° - a 2 crn + 2 abcr n + b 2 o * 2 2 

^ 22 J L^J 

The preceding results can be extended to a linear combination of p random variables: 

The linear combination c'X = CiXy +-1- c p X p has 

mean = £(c'X) = c'/* 

variance = Var (c'X) = c'Xc (243) 

where = £(X) and X = Cov(X). 


In general, consider the q linear combinations of the p random variables 
X } , ...» X p \ 

Zi = c n X i + c n X 2 + *** + 

Z 2 = c 21 X 1 + c 22 X 2 + ■ • ■ + c 2 p x p 


Zq ~ Cq\X\ + C q 2 X 2 H-+ C qp X p 

Z\ c n c i2 -*■ ci p X } 

z 2 C21 C 2 2 **• C 2 p X 2 


' Zl" 
Z 2 


1- 

'x N ... 
W 1_ 



c q\ c q2 

(qxp) 


The linear combinations Z = CX have 

Mz - E( Z) - £(CX) = C/tx 
X z = Cov(Z) = Cov(CX) = CX x C' 


where /l* x and X x are the mean vector and variance-covariance matrix of X, respec 
tively, (See Exercise 2.28 for the computation of the off-diagonal terms in CX x C'.) 

We shall rely heavily on the result in (2-45) in our discussions of principal com 
ponents and factor analysis in Chapters 8 and 9. 


Example 2.15 (Means and covariances of linear combinations) Let X' = [X it X 2 ] 
be a random vector with mean vector *i x = [p \, p^] and variance-covariance matrix 
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Find the mean vector and covariance matrix for the linear combinations 

= X l - X 2 
Z 2 = X 2 + X 2 


in terms of /l*x and X x , 
Here 


Mz — £(Z) = C/t x = 


Mi M 2 
Ml + M2_ 


2 Z = Cov(Z) = C2 X C' = 


0*11 o 12 

0*12 °22 


= °“n 2 o * 12 + 0*22 o- xl - 0*22 

o-n - a 22 . a n + 2a n + a 22 _ 

temis^X^ vanish T^-Th 113 * 1S,lf ^ and Xl have e£ l ual variances—the off-diagonal 
ence of two random variaW^^^^id^riral'varian^a^imc<MTrelat«t ^ 


Partitioning the Sample Mean Vector 
and Covariance Matrix 

n observations on p variable! averages constructed flora 


’ ( x n ~ x i) 2 


l n 

n 2 (Xji ~ x i)(Xj P ~ x p ) 


1 n 

-^( x , i - Xi )( x jp - x p ) 


n 2 ( X JP ~ x p f 


be the corresponding sample variance-covariance matrix. 



2 7 Matrix Inequalities and Maximization 

Maximization principles play an important role in several multivariate techniques. 
Linear discriminant analysis, for example, is concerned with allocating observations 
to predetermined groups. The allocation rule is often a linear function of measure¬ 
ments that maximizes the separation between groups relative to their within-group 
variability. As another example, principal components are linear combinations of 
measurements with maximum variability. 

The matrix inequalities presented in this section will easily allow us to derive 
certain maximization results, which will be referenced in later chapters. 

Cauchy-Schwarz Inequality. Let b and d be any two p x 1 vectors. Then 

(b'd) 2 s (b'b) (d'd) 

with equality if and only if b = cd (or d = cb) for some constant c. 


(2-48) 
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Proof. The inequality is obvious if either b = 0 or d = 0. Excluding this possibility, 
consider the vector b - xd, where x is an arbitrary scalar. Since the length of 
b - xd is positive for b - xd ^ 0, in this case 

0 < (b - xd)'(b - xd) = b'b - xd'b - b'(xd) 4- x 2 d'd 
= b'b - 2x(b'd) 4- x 2 (d'd) 

The last expression is quadratic in x. If we complete the square by adding and 
subtracting the scalar (b'd) 2 /d'd, we get 

(b'd) 2 (b'd) 2 

0 < b'b - - 2*(b'd) + * 2 (d'd) 

(b'd) 2 ( b'd V 

" b '„ - W- + <«>(* " 

The term in brackets is zero if we choose x = b'd/d'd, so we conclude that 

(b'd) 2 

(Xb'b-^f 

or (b'd) 2 < (b'b)(d'd) if b ^ xd for some x. 

Note that ifb = cd, 0 = (b - cd)'(b - cd), and the same argument produces 
(b'd) 2 = (b'b) (d'd). ■ 

A simple, but important, extension of the Cauchy-Schwarz inequality follows 
directly. 

Extended Cauchy-Schwarz Inequality. Let b and d be any two vectors, and 

let B be a positive definite matrix. Then (pX1) (/7X1 ^ 

(pXp) 

(b'd) 2 s (b'Bb)(d'B _1 d) (2-49) 

with equality if and only if b = cB“*d (or d = cBb) for some constant c. 

Proof. The inequality is obvious when b = 0 or d = 0. For cases other than these, 
consider the square-root matrix B 1/2 defined in terms of its eigenvalues A ; and 

p 

the normalized eigenvectors e,- as B 1/2 = ^ e,e' . If we set [see also (2-22)] 


it follows that 


b -!/2 = ^ _L e e ; 

h vi;" 


b'd = bid = b'B I/2 B“ I/2 d = (B 1/2 b)'(B 1/2 d) 


and the proof is completed by applying the Cauchy-Schwarz inequality to the 
vectors (B 1/2 b) and (B“ 1/2 d). ■ 


The extended Cauchy-Schwarz inequality gives rise to the following maximiza¬ 
tion result. 
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, 1 ,t B be positive definite and d be a given vector. 

Maximization Lemma. Let ( oe P osu <p*D 

Then, for an arbitrary nonzero vector 

max^——= d'B _, d ( 2 - 50 ) 

xlo \'Bx 

• j i _ _ - = r u _1 d for any constant c =? 0. 

with the maximum attained whe (pxp)(pxi) 

Pit - d'B'M 
x'Bx 

„„„ v oives Eauation (2-50) because the bound is attained for 
Taking the maximum over x gives tq v ; — 

x = cB^d. 

A final maximization result will provide us with an interpretation of eigenvalues 

Maximization of Quadratic Forms for Pointsjm the Let 

positive definite matrix with eigenvalues A t - 2 p 

normalized eigenvectors e,. «*. • • • > V 111611 

max^f = A, (attained when x = «i) (2 _ 51) 


min = A p (attained when x = e p ) 

x'x 


Moreover, 


x'Bx _ (attained whenx — e*+i, k - 1,2,.. •, P 1 ) (£52) 

* lei,- • XX 


where the symbol X is read “is perpendicular to.” 


Proof. Let P be the orthogonal matrix whose columns are the eigenvectors 
. ‘and A be the diagonal matrix with eigenvalues A, A 2 .A p along the 

ssksSSM*>- i- <^>i - «,?„ ■ 

Consequently, x * «implies y * »• T 1 ™ 8 - 

xBx _ x PA 1/2 P PA 1/2 P * _ y^y 

= xppx y'y yy 


i=l--sA^ 

iw 2^ 


(2-53) 
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Setting x = ej gives 


since 



For this choice of x, we have y' Ay/y'y = A^l = Ai, or 

r *' Ho 

= e',Be, = A, (2-54) 

e l e l 

A similar argument produces the second part of (2-51). 

Now,x = Py = + ^e 2 + * • • 4- y p e p , sox ± ei,..., e k implies 

0 = ejx - yieje! + y 2 e , i e 2 + * * • + y p e}e p = y h i ^ k 

Therefore, for x perpendicular to the first k eigenvectors e*, the left-hand side of the 
inequality in (2-53) becomes 

x , Bx E a -l 2 

x Bx _ i=k+l 

X X ^ 2 

2 yt 

i=k+l 


Taking y k + x = 1, y k + 2 = ■ * • = y p = 0 gives the asserted maximum. 


For a fixed x 0 ¥= 0, xqBx^/xqx 0 has the same value as x'Bx, where 
x' = Xq/Vx^Xo i s of unit length. Consequently, Equation (2-51) says that the 
largest eigenvalue, A 1? is the maximum value of the quadratic form x'Bx for all 
points x whose distance from the origin is unity. Similarly, \ p is the smallest value of 
the quadratic form for all points x one unit from the origin.The largest and smallest 
eigenvalues thus represent extreme values of x'Bx for points on the unit sphere. 
The “intermediate” eigenvalues of the p X p positive definite matrix B also have an 
interpretation as extreme values when x is further restricted to be perpendicular to 
the earlier choices. 





Vectors and Matrices: 
Basic Concepts 


Vectors 


Many concepts, such as a person’s health, intellectual abilities, or personality, cannot 
be adequately quantified as a single number. Rather, several different measure¬ 
ments x x ,x 2 ,. .., are required. 

Definition 2 A. I . An m-tuple of real numbers (x 1 ,x 2 >-. -,x n ...,x m ) arranged in a 
column is called a vector and is denoted by a boldfaced, lowercase letter. 

Examples of vectors are 


Vectors are said to be equal if their corresponding entries are the same. 


Definition 2A.2 (Scalar multiplication). Let c be an arbitrary scalar. Then the 
product cx is a vector with /th entry ex,-. 

To illustrate scalar multiplication, take c Y = 5 and c 2 = -1.2. Then 


10 and c 2 y = (-1.2) 
-10 


= 5 2 = 
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Definition 2A.3 (Vector addition). The sum of two vectors x and y, each having the 
same number of entries, is that vector 

% = x + y with ith entry = *,• + y t 

Thus, 



x + y = z 


Taking the zero vector, 0, to be the m-tuple (0,0,..., 0) and the vector -x to be the 
m-tuple (-*!, -x 2 , ..., -* m ), the two operations of scalar multiplication and 
vector addition can be combined in a useful manner. 


Definition 2A.4. The space of all real m-tuples, with scalar multiplication and 
vector addition as just defined, is called a vector space . 

Definition 2A.5. The vector y = a 1 x 1 + a 2 x 2 + ■ • ■ + is a linear combination of 
the vectors x 2 , x 2 ,..., x*. The set of all linear combinations of x x , x 2 , ..., x* , is called 
their linear span. 

Definition 2A.6. A set of vectors x 2 , x 2 ,..., x k is said to be linearly dependent if 
there exist k numbers (a { , a 2 ,..., a k ), not all zero, such that 

aiX! + a 2 x 2 + • ■■ + a k x k = 0 

Otherwise the set of vectors is said to be linearly independent. 


If one of the vectors, for example, x f , is 0, the set is linearly dependent. (Let a t be 
the only nonzero coefficient in Definition 2A.6.) 

The familiar vectors with a one as an entry and zeros elsewhere are linearly 
independent. For m — 4, 



so 


0 = a x Xi + a 2 x 2 + a 3 x 3 + a 4 x 4 


a 1 • 1 + a 2 • 0 + a 3 • 0 + a 4 • 0 

• 0 + a 2 • 1 + a 3 • 0 + a 4 • 0 

• 0 + a 2 • 0 + a 3 • 1 + a 4 • 0 

ai' 0 + a 2 • 0 + a 3 • 0 + a 4 • 1 



implies that a x = a 2 - a 3 — a 4 — 0. 
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As another example, let ft - 3 and m = 3, and let 


1 , *2 


111611 2xi - x 2 + 3x 3 = 0 

_ v are a linearly dependent set of vectors, since any one can be written 

’ 2X1 * ** 

2A.7. Any « of m linearly i»dep«,d«nt vector k o.llod . i mU for tho 
vector space of all m-tuples of real numbers. 

Result 2A.I. Every vector can be expressed as a unique linear combination ofa 
fixed basis. 


With m = 4 , the usual choice of a basis is 


These four vectors were shown to be linearly independent. Any vector * can be 
uniquely expressed as 


1 0 0 

+ *2 0 + *3 1 + * 4 0 


., „ elements may be regarded geometrically as a point in 
A vector consisting of m ^ W J m = £ the vector x may be regarded as 

m dime nbnethe S point in the plane with coordinates x, and x 2 . 

"^Vectors have the geometrical properties of length and direction. 



Definition 2A.8. The length of a vector of m elements emanating from the origin is 

given by the Pythagorean formul a: _ 

length of x = L x = V*i + xi + ■ • ■ + 
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Definition 2A.9. The angle 6 between two vectors x and y, both having m entries, is 
defined from 

{x x y x + x 2 y2 + ■■ + x m y m ) 
cos( 0 ) =- —- - 

L, X L-,y 

where L x = length of x and L y ~ length of y, x\ 2 * 2 , • • •, are the elements of x, 
and y x , 3 ^ 2 ,..., y m are the elements of y . 



Definition 2A.I0. The inner (or dot) product of two vectors x and y with the same 
number of entries is defined as the sum of component products: 

+ *2 y 2 + " * + Xmy m 

We use the notation x'y or y'x to denote this inner product. 

With the x'y notation, we may express the length of a vector and the cosine of 
the angle between two vectors as 
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Definition 2A. 11 . When the angle between two vectors x, y is 6 = 90° or 270°, we 
say that x and y are perpendicular. Since cos(0) = 0 only if 0 = 90° or 270°, the 
condition becomes 

x and y are perpendicular if x'y = 0 

We write xly.. 


The basis vectors 


are mutually perpendicular. Also, each has length unity. The same construction 
holds for any number of entries m. 

Result 2A.2, 

(a) z is perpendicular to every vector if and only if z = 0 . 

(b) If z is perpendicular to each vector X] ,x 2 ,.then z is perpendicular to 
their linear span. 

(c) Mutually perpendicular vectors are linearly independent. ■ 


Definition 2A. 1 2 . The projection (or shadow) of a vector x on a vector y is 

(x'y) 

projection of x on y = — y~ y 
L y 

If y has unit length so that L y = 1, 

projection of x on y = (x'y)y 

If yj, y 2 ,.. •, y r are mutually perpendicular, the projection (or shadow) of a vector x 
on the linear span of y h y 2 ,..., y r is 

(*'yi) , (x'yz) , (*'yr) 

— yi + — y2 + -. + — y . 


Result 2A.3 (Gram-Schmidt Process). Given linearly independent vectors xj, 
x 2 ,..., x k , there exist mutually perpendicular vectors iij, u 2 , ..., u k with the same 
linear span. These may be constructed sequentially by setting 
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We can also convert the u’s to unit length by setting z, = uy/Vuju). In this 

k-1 

construction, (x k z j) z j is the projection of x k on Zj and ^ (x*z y -)zy is the projection 
of x k on the linear span of x,, x 2 ,..., x*.,. * m 

For example, to construct perpendicular vectors from 


we take 


U] = X] = 

0 

_ 2 _ 

u[u l = 4 2 + 0 2 + 0 2 + 2 2 = 20 
x 2 u i = 3(4) + 1(0) + 0(0) - 1(2) = 10 


and Zj 


1 0 

V20 0 ’ 


V6 0 

-2 


Matrices 


Definition 2A.I3. An m X k matrix , generally denoted by a boldface uppercase 
letter such as A, R, S, and so forth, is a rectangular array of elements having m rows 
and A: columns. 

Examples of matrices are 


-7 2 
A = 0 1 

3 4 


1 .7 —.3 

X = .7 2 1 

-.3 1 8 


4 -2 1/x 

E = [e,] 


r 1 

0 

o" 

, I = 0 

1 

0 

J L° 

0 

1_ 
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In our work, the matrix elements will be real numbers or functions taking on values 
in the real numbers. 

Definition 2A. 14. The dimension (abbreviated dim) of an m X k matrix is the ordered 
pair (m> k)\ m is the row dimension and k is the column dimension.The dimension of a 
matrix is frequently indicated in parentheses below the letter representing the matrix. 
Thus, the m X k matrix A is denoted by A . 

(mXJt) 

In the preceding examples, the dimension of the matrix X is 3 X 3, and this 

information can be conveyed by writing X . 

‘ (3X3) 

An m X k matrix, say. A, of arbitrary constants can be written 


fill 

a \2 ' 

*' a lk 

A = 021 

a 22 

‘ * a 2k 

[mxk) : 



_^ml 

a m2 * 

” a mk 


or more compactly as A = {«/,}, where the index i refers to the row and the 

(mXk) 

index j refers to the column. 

An m X 1 matrix is referred to as a column vector . A 1 x k matrix is referred 
to as a row vector. Since matrices can be considered as vectors side by side, it is nat¬ 
ural to define multiplication by a scalar and the addition of two matrices with the 
same dimensions. 

Definition 2A. 15. TVo matrices A = {a, y } and B = {6 ty } are said to be equal, 

written A = B, if a i} = 6 iy , i = 1,2,..., m, j = 1,2,..., k. That is, two matrices are 
equal if 

(a) Their dimensionality is the same. 

(b) Every corresponding element is the same. 

Definition 2A.I6 (Matrix addition). Let the matrices A and B both be of dimension 
m X k with arbitrary elements a, y and 6 iy) i = 1,2,..., m, j — 1,2respec¬ 
tively. The sum of the matrices A and B is an m x k matrix C, written C = A + B, 
such that the arbitrary element of C is given by 

Cij &ij F bij i 1 , 2 ,..., m, j — 1 , 2 ,...,/: 

Note that the addition of matrices is defined only for matrices of the same 
dimension. 



For example, 
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Definition 2A. 1 7 (Scalar multiplication). Let c be an arbitrary scalar and A «= {a,. 

(mxt) ' 

Then cA = Ac = B = {b i} }, where b ti = ca u ~ a u c, i = l,2,. ..,m, 

(mxfc) (mx*) ( m xk) tJ lJ tJ 3 

7 = 1,2 


Multiplication of a matrix by a scalar produces a new matrix whose elements are 
the elements of the original matrix, each multiplied by the scalar. 

For example, if c = 2, 



Definition 2A. 18 (Matrix subtraction). Let A = {a i} ) and B = {b i} } be two 

(/nXit j 

matrices of equal dimension.Then the difference between A and B, written A — B, 
is an m X k matrix C = {c /y } given by 

C — A — B = A+ (—1)B 

That is, aj = a i} + (-1)6, y = a u - b i} , i = 1, 2 , ..., m, j = 1, 2 , ..., k. 

Definition 2A. 19. Consider them x k matrix A with arbitrary elements a ij9 i = 1 , 
2,...,m, j = 1,2,...,*. The transpose of the matrix A, denoted by A', is 
the k X m matrix with elements a jiy j = 1, 2, ..., k 7 i = 1, 2, ..., m. That is, the 
transpose of the matrix A is obtained from A by interchanging the rows and 
columns. 


As an example, if 



Result 2A.4* For all matrices A, B, and C (of equal dimension) and scalars c and d , 
the following hold; 

(a) (A + B) + C = A + (B + C) 

(b) A + B = B + A 

(c) c(A + B) ~ cA + cB 

(d) (c + d )A = cA + dA 

(e) (A + B) ; = A' + B' 

(f) (cd)A = c(dA) 

(g) (cA)' = cA' 


(That is, the transpose of the sum is equal to the 
sum of the transposes.) 
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Definition 2A.20. If an arbitrary matrix A has the same number of rows and columns, 
then A is called a square matrix. The matrices 2, I, and E given after Definition 2AJ3 
are square matrices. 


Definition 2A.21. Let A be a k X k (square) matrix.Then A is said to be symmetric 
if A = A'. That is, A is symmetric if a,j = a jh i = 1,2,..., k> j = 1,2,..., k. 


Examples of symmetric matrices are 


I 

(3X3) 


] 0 0 
0 1 0 
0 0 1 



a c e f 

B = c b g d 

(4x4) e g c a 

J d a d_ 


Definition 2A.22. The k X k identity matrix, denoted by I , is the square matrix 

with ones on the main (NW-SE) diagonal and zeros elsewhere. The 3X3 identity 
matrix is shown before this definition. 


Definition 2A.23 (Matrix multiplication). The product AB of an m X n matrix 
A = {a,;} and an n X k matrix B = {b, y } is the m X k matrix C whose elements 
are 

n 

c U~^ a i( b ej 1 = 1,2,..., m 7 = 1 , 2 ,...,/: 

l-l 

Note that for the product AB to be defined, the column dimension of A must 
equal the row dimension of B. If that is so, then the row dimension of AB equals 
the row dimension of A, and the column dimension of AB equals the column 
dimension of B. 


For example, let 


Then 




(2X3) (3X2) 


11 

20 " 


c n 

c \2 

_32 

31 _ 


__ c 21 

C 22~ 


(2X2) 


where 
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Cu = (3)(3) + (—1)(6) + (2) (4) = 11 
Ci2 = (3)(4) + (—1)( —2) + (2)(3) = 20 
c 2] = (4)(3) + (0) (6) + (5) (4) = 32 
c 22 = (4)(4) + (0)(—2) + (5)(3) = 31 
As an additional example, consider the product of two vectors. Let 



Then x' = [1 0 “2 3] and 



2~ 



r 11 

x'y = [1 0 -2 3] 

-3 

-1 

= [-20] = [2 -3 -1 

-8] 

O Cn) 

1 


_-8_ 



L 3 J 


Note that the product xy is undefined, since x is a 4 X 1 matrix and y is a 4 X 1 ma¬ 
trix, so the column dim of x, 1, is unequal to the row dim of y, 4. If x and y are vectors 
of the same dimension, such as n X 1, both of the products x'y and xy' are defined. 
In particular, y'x = x'y = Xiy\ 4- x 2 y 2 + ■•• + x„y H9 and xy' is an n X n matrix 
with ij th element jc z y y . 

Result 2A.5. For all matrices A, B, and C (of dimensions such that the indicated 
products are defined) and a scalar c, 

(a) c(AB) = (cA)B 

(b) A(BC) = (AB)C 

(c) A(B + C) = AB + AC 

(d) (B + C)A = BA + CA 

(e) (AB)' = B'A' 

More generally, for any x ; such that Ax y is defined, 

(f) 2 Ax y = A 2 x j 

J - 1 /=! 

(g) 2 (AXjKAXy)' = A (2 XyXj) A' 

M \ j=1 / ■ 
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There are several important differences between the algebra of matrices and 
the algebra of real numbers. T\vo of these differences are as follows: 

1. Matrix multiplication is, in general, not commutative. That is, in general, 
AB 9 * BA. Several examples will illustrate the failure of the commutative law 

(for matrices). 


is not defined. 


0 3 -1 
2 4 7 


9 10 
35 33 


1 0 1 
2-3 6 


19 -18 43 
= -1-3 3 

10 -12 26 


2 14 -1 


8 -1 
-12 7 


2. Let 0 denote the zero matrix, that is, the matrix with zero for every element. In 
the algebra of real numbers, if the product of two numbers, ab , is zero, then 
a = 0 or b = 0. In matrix algebra, however, the product of two nonzero main- 
ces may be the zero matrix. Hence, 

AB = 0 

(mxn)(uxit) (mxk) 

does not imply that A = 0 or B = 0. For example, • 


3 1 3 
1 2 2 


It is true, however, that if either 

A B = 0 

(mxn)(nxk) (mXk) 


A = 0 

(mXn) ( mXn ) 
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Definition 2A.24* The determinant of the square k X k matrix A = {a,y}, denoted 
by IA |, is the scalar 

| A | = a,, ifyfc = l 

|A|= 2*1/I A 1/I(-1) 1+ ' if At > 1 
/-1 

where A ly - is the (k - 1) X (k - 1) matrix obtained by deleting the first row and 

k 

;th column of A. Also, | A [ = ^ a { j | A iy |(-l) i+; , with the ith row in place of the first 

M 

row. 

Examples of determinants (evaluated using Definition 2A.24) are 

l ! = 1141 ( 1 ) 2 + 3|6|(—l ) 3 = 1(4) + 3(6)(—1) = —14 


In general, 

*n *12 
*21 ^22 


a \ l a 22 ( I ) 2 + 012 fl 2 l (“ l ) 3 - a \\^22 ~ a \ 2 a 2 \ 


3 1 6 

7 4 5 =3 

2 -7 1 


= 3 |-7 l \^ +1 \ 2 l | t “ 1 ) 3 + 6 | 

= 3(39) - l(-3) + 6(-57) = -222 


100 

0 1 0 =lj (-l ) 2 + o J 0 (_i)3 + 0 ° J (-!)«-1(1) - 1 
0 0 1 


If I is the/: X k identity matrix, |I| = 1. 


*11 

*12 

*13 

*21 

*22 

*23 

*31 

*32 

*33 

= a 

*22 
11 *32 

*23 

*33 


(- 1) 2 + a l 


a 21 a 23 

a 3l a 33 


(-1) 3 + «t: 


^21 fl 22 
fl 3 l a 32 


— a ll a 22 a 33 + a 12 a 23 a 3i + a 21 a 32 a l3 ~ a 3l a 22 a 13 ~ a 2l a 12 a 33 ~ a 32 a 23 a ll 

The determinant of any 3x3 matrix can be computed by summing the products 
of elements along the solid lines and subtracting the products along the dashed 
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lines in the following diagram. This procedure is not valid for matrices of higher 
dimension, but in general. Definition 2A.24 can be employed to evaluate these 
determinants. 



Definition 2A.25. The row rank of a matrix is the maximum number of linearly inde¬ 
pendent rows, considered as vectors (that is, row vectors). The column rank of a matrix 
is the rank of its set of columns, considered as vectors. 


For example, let the matrix 



The rows of A, written as vectors, were shown to be linearly dependent after 
Definition 2A.6. Note that the column rank of A is also 2, since 



but columns 1 and 2 are linearly independent. This is no coincidence, as the 
following result indicates. 


Result 2A.6. The row rank and the column rank of a matrix are equal. 
Thus, the rank of a matrix is either the row rank or the column rank. 


Vectors and Matrices; Basic Concepts 95 


Definition 2A.26. A square matrix A is nonsingular if A x = 0 implies 

(kxk) (kxk){kxi) (*xi) 

that ^x^ = ^0 . If a matrix fails to be nonsingular, it is called singular. Equivalently, 

a square matrix is nonsingular if its rank is equal to the number of rows (or columns) 
it has. 

Note that Ax = jqaj + x 2 a 2 + * * * + Je*a*, where a z is the ith column of A, so 
that the condition of nonsingularity is just the statement that the columns of A are 
linearly independent. 

Result 2A.7. Let A be a nonsingular square matrix of dimension k X £.Then there 
is a unique kxk matrix B such that 

AB = BA = I 

where I is the k X k identity matrix. ■ 

Definition 2A.27. The B such that AB = BA = I is called the inverse of A and is 
denoted by A" 1 . In fact, if BA = I or AB = I, then B = A“\ and both products 
must equal I. 

For example, 



Result 2A.8, 

(a) The inverse of any 2x2 matrix 



A = 


a \\ a 12 ^13 

a 2 \ a 22 a 2 3 

a 31 a 32 ^33 
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is given by 


- A -1 



*22 

*23 

*32 

*33 

*21 

*23 

*3) 

*33 

*21 

*22 

*31 

*32 


l*l 2 

*13 

*32 

*33 

*11 

*13 

*31 

*33 

*11 

*12 

*31 

*32 


*12 

*13 

*22 

*23 

*11 

*13 

*21 

*23 

*11 

*12 

*21 

*22 



In both (a) and (b), it is dear that | A | * 0 if the inverse is to exist. 

(c) In general, A -1 has j, ith entry [| A, 7 |/|A|](-1) <+ ', where A f/ is the matrix 
obtained from A by deleting the ith row and yth column. ■ 

Result 2A.9. For a square matrix A of dimension k X k, the following are equivalent: 

(a) A x = 0 implies x = 0 (A is nonsingular). 

1 (*x*)(*xi) (*X1) (*X1) (*X1) 

(b) | A | # 0. 

(c) There exists a matrix A 1 such that A A 1 = A 'A = I . ■ 

' 7 (kxk) 

Result 2A JO. Let A and B be square matrices of the same dimension, and let the 
indicated inverses exist. Then the following hold: 

(a) (A -1 )' = (A') 1 

(b) (AB ) -1 = BA 1 ■ 



The determinant has the following properties. 


Result 2AJ I. Let A and B be k X k square matrices. 

(a) | A | = | A' | 

(b) If each element of a row (column) of A is zero, then | A | = 0 

(c) If any two rows (columns) of A are identical, then | A | = 0 

(d) If A is nonsingular, then | A | = 1/| A 1 1; that is, | A || A 1 1 = 1. 

(e) | AB | = | A || B | 

(f) | cA | = c k | A |, where c is a scalar. 


You are referred to [ 6 ] for proofs of parts of Results 2A.9 and 2A.11. Some of 
these proofs are rather complex and beyond the scope of this book. ■ 

Definition 2A.28. Let A = be a kxk square matrix. The trace of the matrix A, 

k 

written tr (A), is the sum of the diagonal elements; that is, tr (A) = 2 a u * 
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Result 2AJ2. Let A and B be k X k matrices and c be a scalar. 

(a) tr(cA) = c tr(A) 

(b) tr (A ± B) = tr (A) ± tr(B) 

(c) tr (AB) = tr(BA) 

(d) tr(B - 1 AB) = tr (A) 

(e) tr(AA') =22 a h 

1=1 y=i 


Definition 2A.29. A square matrix A is said to be orthogonal if its rows, considered 
as vectors, are mutually perpendicular and have unit lengths; that is, AA' = I. 


Result 2 A J 3. A matrix A is orthogonal if and only if A -1 = A'. For an orthogonal 
matrix, AA' = A'A = I, so the columns are also mutually perpendicular and have 
unit lengths. ■ 

An example of an orthogonal matrix is 

2 2 2 

2 2 2 

1 _ 1 1 

2 2 2 

1 1 

2 2 2 _ 

Clearly, A = A', so A A' = A'A = AA. We verify that AA - I = AA' = A'A, or 

I I I i~| r i l l l 

2222 2222 

1 _ 1 1 1 1 __I 1 1 

2222 2 "2 22 

1 1 _ 1 1 1 1 _1 1 

2222 2222 

1 I I _I ill _I 

2 2 2 2JL2 2 2 2 


so A' = A -1 , and A must be an orthogonal matrix. 

Square matrices are best understood in terms of quantities called eigenvalues 
and eigenvectors. 

Definition 2A.30. Let A be a k X k square matrix and I be the kxk identity ma¬ 
trix. Then the scalars Aj, A 2 ,..., A* satisfying the polynomial equation | A — Al | = 0 
are called the eigenvalues (or characteristic roots) of a matrix A. The equation 
| A — AI | = 0 (as a function of A) is called the characteristic equation. 



"l 

0 

0 

0 


0 

1 

0 

0 


0 

0 

1 

0 


_0 

0 

0 

1_ 


I 



For example, let 


A = 


1 0 
1 3 
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IA - All 


1 ol.fi 0 
1 3 J L° 1 


1 - A 0 
1 3 - A 


(1 - A) (3 - A) =0 


implies M there me -o reels, A, - 1 At - 3. The eige.tv.lues of A =.e 3 
and 1. Let 

‘13-4 2 

A- -4 13 -2 

2 -2 10. 

Then the equation 

, A _ AI | = D 4 13 - A —2 — -A 3 + 36A 2 - 405A + 1458 = 0 
2 -2 10 - A 

has three roots: A, = 9, A 2 = 9, and A, = 18; that *9,9, and 18 are the eigenvalues 
of A. 

Definition 2A.31- Let A be a square matrix of dimension k x k and let A be an eigen¬ 
value of A. If * is a nonzero vector ( J such that 

Ax = Ax 

then x is said to be an eigenvector (characteristic vector) of the matrix A associated with 
the eigenvalue A. 

An equivalent condition for A to be a solution of the eigenvalue-eigenvector 
equation fa | A - AI| = 0. This follows because the statement that Ax Ax fo 

some A and x # 0 implies that 

0 = (A - AI)x = Xi coli(A - AI) +•" + ■** col k (A - AI) 

That is the columns of A - AI are linearly dependent so, by Result 2A.9(b), 

| A - AI | = 0, as asserted. Following Definition 2A.30, we have shown that th 

eigenvalues of r 

are A = 1 and A 2 = 3. The eigenvectors associated with these eigenvalues can be 
determined by solving the following equations: 


1 0 x, 
.1 3_ L *2 
Ax = AjX 


X 

= 1 

J L x 
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1 0 

1 3_ x 2 
Ax = A 2 x 


From the first expression, 


*i = x x 

x x + 3x 2 = x 2 
or 

xj = —2x2 

There are many solutions for x x and x 2 . 

Setting x 2 = 1 (arbitrarily) gives x x = -2, and hence, 


is an eigenvector corresponding to the eigenvalue 1. From the second expression, 

x x + 3x 2 = 3x 2 

implies that x x = 0 and x 2 = 1 (arbitrarily), and hence. 


is an eigenvector corresponding to the eigenvalue 3. It is usual practice to determine 
an eigenvector so that it has length unity. That is, if Ax = Ax, we take e - x/vfx 
as the eigenvector corresponding to A. For example, the eigenvector for A x = 1 is 
ei = [-2/V5, 1/V5]. 

Definition 2A.32. A quadratic form Q(x) in the k variables x x , x 2 , ..., x k is Q(x) = x'Ax, 
where x' = [x x , x 2 ,..., x k ] and A is a £ x k symmetric matrix. 


Note that a quadratic form can be written as Q{x) = X 2 a ij x i x r F° r exam P le > 

'=i 7*1 


G(x) = [*1 x 2. 


11 x x _ 

11 x 2 


x\ + 2x 1 x 2 + x\ 


1 3 0 x x 

G( x ) = \ x i x 2 x $] 3 -1 -2 x 2 = x\ + 6x x x z - x\~ 4x 2 x 3 + 2x\ 

_0 -2 2 _ 

Any symmetric square matrix can be reconstiuctured from its eigenvalues 
and eigenvectors. The particular expression reveals the relative importance of 
each pair according to the relative size of the eigenvalue and the direction of the 
eigenvector. 
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Result 2A./4. The Spectral Decomposition. Let A be a A: X k symmetric matrix. 
Then A can be expressed in terms of its k eigenvalue-eigenvector pairs (A,-, e, ) as 

k 

A = 2 A < e i e ! ■ 

i=l 


For example, let 


2.2 .4 

.4 2.8 


|A - Al| = A 2 - 5A + 6.16 - .16 = (A - 3)(A — 2) 

so A has eigenvalues Ai = 3 and A 2 = 2. The corresponding eigenvectors are 
ej = [l/V5,2/V5] and e 2 = [2/V5, -l/v3], respectively. Consequently, 


2.2 .4 

.4 2.8 


1 2 


VE 2 -l 
-l \_V5 V5 
V5 


VO 

l_ 

1 .2*1 

1.6 

1 

bo 

[1.2 

2 . 4 J + 

-.8 



The ideas that lead to the spectral decomposition can be extended to provide a 
decomposition for a rectangular, rather than a square, matrix. If A is a rectangular 
matrix, then the vectors in the expansion of A are the eigenvectors of the square 
matrices AA' and A'A. 

Result 2A. 15. Singular-Value Decomposition . Let A be an m X k matrix of real 
numbers. Then there exist an m X m orthogonal matrix U and a k X k orthogonal 
matrix V such that 

A = UAV' 

where the m X k matrix A has (*, i) entry A,- ^ 0 for / = 1,2,..., min(m, k) and the 
other entries are zero. The positive constants A, are called the singular values of A. ■ 

The singular-value decomposition can also be expressed as a matrix expansion 
that depends on the rank r of A. Specifically, there exist r positive constants 
Ai, A2, •.., A r , r orthogonal m X 1 unit vectors iq, 112 ,..., u r , and r orthogonal 
k X Lunit vectors , v 2 , • • • > v r , such that 


A = 2 AjUjVj = U r A f V; 

l‘ = l 

where U r = [uj, u 2 ,..., u r ], V r = [v*, v 2 ,..., v f ], and A r is an r x r diagonal matrix 
with diagonal entries A, . 
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Here AA' has eigenvalue-eigenvector pairs (A 2 , u,), so 

AA'u, = u, 


with A 2 , A 2 ,..., A 2 > 0 = A, +1 , A? +2 , •.., A?„ (for m > k).Then = A^A'u,-. Alter¬ 
natively, the v,- are the eigenvectors of A'A with the same nonzero eigenvalues A/ . 

The matrix expansion for the singular-value decomposition written in terms of 
the full dimensional matrices U, V, A is 

A — U A V' 

(mXk) (mXm)(mXk)(kXk) 

where U has m orthogonal eigenvectors of AA' as its columns, V has k orthogonal 
eigenvectors of A'A as its columns, and A is specified in Result 2A.15. 

For example, let 


3 1 1 

-1 3 1_ 


3 1 1 

-1 3 1 


11 1 
1 11 


Y° u may verify that the eigenvalues y = A 2 of AA' satisfy the equation 
y ~ 22y + 120 = (y — 12) (y — 10), and consequently, the eigenvalues are 
7 i = = 12 and y 2 = A 2 = 10. The corresponding eigenvectors are 


, r 1 1 

= V! 


1 -1 

V5 vf * respectively. 


3 -1 

A'A = 1 3 

1 1 


3 1 1 
-1 3 1 


10 0 . 2 

= 0 10 4 


so | A'A - yl| = “y 3 - 22y 2 - 120y = -y(y - 12) (y - 10), and the eigenvalues 
are yj = \\ = 12, y 2 = A 2 = 10, and y 2 ~ A 2 = 0. The nonzero eigenvalues are the 
same as those of AA'. A computer calculation gives the eigenvectors 

, __ j_ 21 , _ 2 -1 1 r 1 2 - 5 ” 

Vl_ Lv6 VZ V5J V2- Lv! v! °J’ and ^ = Lv55 VW V35_' 

Eigenvectors Vj and v 2 can be verified by checking: 

r>° ° 2 i 1T 1 ! 1 pi 

A'Av,- 0 10 4 -= 2 \-12— 2 - »iv, 


are y 1 = A 2 - 12, y 2 = A 2 


1 2 1 

V6 V6 V6 


A'Avi = 


10 0 
0 10 
2 4 


2 =12 
1 


10 

0 

2" 

1 

0 

10 

4 

_ 2 

4 

2_ 



-1 =10 
0 


= A 2 v. 


-1 = A 2 v 2 

0 
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Taking Ai = Vl2 and A 2 = VT6, we find that the singular-value decomposition of 



The equality may be checked by carrying out the operations on the right-hand side. 

The singular-value decomposition is closely connected to a result concerning 
the approximation of a rectangular matrix by a lower-dimensional matrix, due to 
Eckart and Young ([2]). If a m X k matrix A is approximated by B, having the same 
dimension but lower rank, the sum of squared differences 

iiK-M 2 = tr [(A’B)(A-Bn 
1 / =1 

Result 2A.16. Let A be an m X k matrix of real numbers with m > k and singular 
value decomposition UAV'. Let s < k = rank(A).Then 

B = 2 A ' u > v ! 

i=l 

is the rank-5 least squares approximation to A. It minimizes 

tr[(A - B)(A - B)'] 

over all m X k matrices B having rank no greater than s. The minimum value, or 

k 

error of approximation, is 2} A*. ■ 

i=s+1 

To establish this result, we use UU' = I m and W' = I* to write the sum of 
squares as 

tr[(A - B) (A - B)'] = tr[UU'(A - B) W'(A - B)'] 

= tr[U'(A - B)W'(A - B)'U] 

= tr[(A - C)(A - C)'] = S 2 (A „ - erf = 2 (A, - c„) a + 2S C U 

i=l /= 1 i=l i*j 

where C = U'BV. Clearly, the minimum occurs when c iy - — Ofori ^ ;andc /( - = A/for 

5 

the s largest singular values. The other c u = 0. That is, UBV' = A 5 or B = 2 A,- u, v*. 


Exercises 103 


Exercises 


2.1. Let x' = [5, 1, 3] and y' = [ —1, 3, 1]. 

(a) Graph the two vectors. 

(b) Find (i) the length of x, (ii) the angle between x and y, and (iii) the projection of y on x. 

(e) Since x = 3 and y = 1, graph [5 - 3,1 - 3,3 - 3] = [2, -2,0] and 
[-1 - 1,3 - 1,1 - 1] = [-2,2,0], 1 

2.2. Given the matrices 


perform the indicated multiplications. 

(a) 5A 

(b) BA 

(c) A'B' 

(d) C'B 

(e) Is AB defined? 

2.3. Verify the following properties of the transpose when 


and C = 


(a) (A')' = A 

(b) (CT 1 = (CT 1 )' 

(c) (AB)' = B'A' 

(d) For general A and B ,(AB)'=B'A' 

(mXA) ( kxt) X ' 

2.4. When A -1 and B 1 exist, prove each of the following 

(a) (A')’ 1 = (A -1 )' 

(b) (AB)" 1 = B-'A- 1 

Hint: Part a can be proved by noting that AA _1 = I, I = I , and (AA“ 
Part b follows from (B-'A'^AB = B -1 (A _I A)B = B -1 B = I. 

2.5. Check that 


(A-‘)'A\ 


Q = 


is an orthogonal matrix. 


2.6. Let 


(a) Is A symmetric? 

(b) Show that A is positive definite. 
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2.7. Let A be as given in Exercise 2.6. 

(a) Determine the eigenvalues and eigenvectors of A. 

(b) Write the spectral decomposition of A. 

(c) Find A -1 . 

(d) Find the eigenvalues and eigenvectors of A -1 . 

2.8. Given the matrix 



find the eigenvalues and A 2 and the associated normalized eigenvectors e! and e 2 . 
Determine the spectral decomposition (2-16) of A. 

2.9. Let A be as in Exercise 2.8. 

(a) Find A -1 . 

(b) Compute the eigenvalues and eigenvectors of A -1 . 

(c) Write the spectral decomposition of A -1 , and compare it with that of A from 
Exercise 2.8. 

2.10. Consider the matrices 


A = 


4 4.001" 

4.001 4.002^ 


and B = 


4 4.001 

4.001 4.002001 _ 


These matrices are identical except for a small difference in the (2,2) position. 
Moreover, the columns of A (and B) are nearly linearly dependent. Show that 
A“ l = (-3)8'*. Consequently, small changes—perhaps caused by rounding—can give 
substantially different inverses. 

2.11. Show that the determinant of the p X p diagonal matrix A = {a, ; } with a, ; = 0, / ^ /, 
is given by the product of the diagonal elements; thus, | A | = a } } a 22 • ■ ■ a pp . 

Hint: By Definition 2A.24, |A| = flnA n + 0 + ••■ + 0. Repeat for the submatrix 
A j! obtained by deleting the first row and first column of A. 

2.12. Show that the determinant of a square symmetric pxp matrix A can be expressed as 
the product of its eigenvalues Aj, A 2 ,..., A p ; that is, | A | = nf~i A,. 

Hint: From (2-16) and (2-20), A = PAP' with P'P = I. From Result 2A.11(e), 
IA | = | P AP' | = | P 11 AP' | = | P 11 A | |P r | = | A | |I |, since |I | = | P'P | = | P' 11 P |. Apply 
Exercise 2.11. 


2.13. Show that | Q | = + 1 or -1 if Q is a p X p orthogonal matrix. 

Hint: |QQ' | = |f |. Also, from Result 2A.11, |Q | |Q' | = |Q | 2 . Thus, | Q | 2 = |I|. Now 
use Exercise 2.11. 

2.14. Show that Q' A Q and A have the same eigenvalues if Q is orthogonal. 

(/>*/>) (/>*/>)(/>*/>) (pXp) 

Hint: Let A be an eigenvalue of A. Then 0 = | A - AJ |. By Exercise 2.13 and Result 
2A.11(e), we can write 0 = |Q' 11A - AI11 Q | — | Q'AQ — AI |, since Q'Q = I. 

2.15. A quadratic form x'A x is said to be positive definite if the matrix A is positive definite. 
Is the quadratic form 3*? + 3x 2 - 2x^2 positive definite? 

2.16. Consider an arbitrary n X p matrix A.Then A'A is a symmetric p X p matrix. Show 
that A'A is necessarily nonnegative definite. 

Hint: Sety = A x so thaty'y = x'A'A x. 
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2.17. Prove that every eigenvalue of a A: x k positive definite matrix A is positive. 

Hint: Consider the definition of an eigenvalue, where Ae = Ae. Multiply on the left by 
e' so that e'Ae = Ae'e. 

2.18. Consider the sets of points (x } , x 2 ) whose “distances” from the origin are given by 

c 2 - 4x\ + 3x 2 - 2V2x,x 2 

for c == 1 and for c 2 = 4. Determine the major and minor axes of the ellipses of con¬ 
stant distances and their associated lengths. Sketch the ellipses of constant distances and 
comment on their positions. What will happen as c 2 increases? 

m 

2 I9 - 1x1 = 2 V^e,e| = PA 1/2 P', where PP' = P'P = I. (The A,’s and the e,’s are 

the eigenvalues and associated normalized eigenvectors of the matrix A.) Show Properties 
(l)-(4) of the square-root matrix in (2-22). 

2.20. Determine the square-root matrix A 1/2 , using the matrix A in Exercise 2.3. Also, deter¬ 
mine A _1/2 , and show that A 1/2 A~ 1/2 = A“ 1/2 A 1/2 = I. 

2.2 I. (See Result 2A.15) Using the matrix 



(a) Calculate A'A and obtain its eigenvalues and eigenvectors. 

(b) Calculate A A' and obtain its eigenvalues and eigenvectors. Check that the nonzero 
eigenvalues are the same as those in part a. 

(c) Obtain the singular-value decomposition of A. 

2.22. (See Result 2A.15) Using the matrix 



(a) Calculate AA' and obtain its eigenvalues and eigenvectors. 

(b) Calculate A'A and obtain its eigenvalues and eigenvectors. Check that the nonzero 
eigenvalues are the same as those in part a. 

(c) Obtain the singular-value decomposition of A. 

2.23. Verify the relationships V^pV 1 / 2 = X and p = (V 1/2 )‘ 1 S(V 1 / 2 )' 1 , where X is the 
P x P population covariance matrix [Equation (2-32)], p is the p X p population cor¬ 
relation matrix [Equation (2-34)], and V ^ 2 is the population standard deviation matrix 
[Equation (2-35)]. 

2.24. Let X have covariance matrix 

"4 0 0" 

2=090 
_0 0 1 . 

Find 

(a) 2- 1 

(b) The eigenvalues and eigenvectors of X. 

(c) The eigenvalues and eigenvectors of X~K 
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2.25. Let X have covariance matrix 



(a) Determine p and V 1 / 2 . 

(b) Multiply your matrices to check the relation V 1/2 pV 1/2 = X. 

2.26. Use X as given in Exercise 2.25. 

(a) Find p, 3 . 

(b) Find the correlation between X x and \ X 2 + \X 3 . 

2.27. Derive expressions for the mean and variances of the following linear combinations in 
terms of the means and covariances of the random variables X 1 ,X 2 , and X 3 . 

(a) X x - 2X 2 

(b) -X Y + 3X 2 

(c) X, + X 2 + X 3 

(e) X x + 2X 2 - X 3 

(f) 3X\ - 4X 2 if X x and X 2 are independent random variables. 

2.28. Show that 

Co \(c ll X l + c 12 ^2 + + C\ p X py c 2l X l + c 22 X 2 + •■* + c 2p X p ) = ciX x c 2 

where cj = [c^ j, c 12 , ..., C\ p ] and c 2 = [c 21 , c 22 , ..., c 2p \. This verifies the off-diagonal 
elements CX x C' in (2-45) or diagonal elements if Ci = c 2 . 

Hint: By (2-43),Zj - E(Z{) - c n (X } - mi) + •*• + c ip (X p - Mp) and 
Z 2 - E(Z 2 ) = c 2 \(Xi — Mi) + *•* + c 2p (X p - Mp)- So Cov(Zj, Z 2 ) = 

E[(Z, - E(Z 1 ))(Z 2 - E (Z 2 ))] = E[(e n (X ! - Mi) + 

+ c\ p (X p - ^ p ))(c 2l (X\ - mi) + c 22 (X 2 - M2) + •** + c 2p (X p - Mp))]- 
The product 

- Ml) + c l2 (^2 - M2) + 

+ c \p(X p ~ Pp))(c 2 i(X l - mi) + ^2(^2 “ M2) + + c 2 p{X p - Mp)) 

= (1 c u(X( - /**)) ( 2 c lm (X m - /*„)) 

= 22 C 1€ C 2 - fLt)(X m ~ Mm) 

(=1 m = l 

has expected value 

S C K c 2m^<m = [C]].Cj p ] X [c 2 ,. C 2p ]'. 

€=*1 m = l 


Verify the last step by the definition of matrix multiplication. The same steps hold for all 
elements. 
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2.29. Consider the arbitrary random vector X' = [X U X 2 , X 3 , X A , X 5 ] with mean vector 
M' = [mi > M 2 ' M 31 M 4 » Ms]- Partition X into 


X<» = x j and X< 2 > = * 4 

L ^ J UJ 

Let X be the covariance matrix of X with general element v ik . Partition X into the 
covariance matrices of X^) and X^ and the covariance matrix of an element of X^ 
and an element of X^. 

2.30. You are given the random vector X' = [X x , X 2 , X 3 , X 4 ] with mean vector 
Mx = [4, 3, 2,1] and variance-covariance matrix 


Partition X as 


A = [1 2] and B 


and consider the linear combinations AX (1) and BX (2) . Find 

(a) £(X<») 

(b) £(AX< ] >) 

(c) Cov(X (,) ) 

(d) Cov (AX* 1 *) 

(e) E(XW) 

(f) £(BX (2) ) 

(g) Cov(X< 2 )) 

(h) Cov (BX< 2) ) 

(i) Cov(Xf I) ,X( 2 )) 

0) Cov(AX< 1 >,BX( 2 )) 

2.31. Repeat Exercise 2.30, but with A and B replaced by 


A = [1 -1] and B = 


2 -1 

0 1 
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2.32. You are given the random vector X' = [A',, X 2 X s \ with mean vector 
Mx “ [2, 4 ’ _1 > 3,0] and variance-covariance matrix 

r 4 —i i -k o" 

-1 3 1-1 o 

2x= 5 1 6 1-1 

-1 140 

0 0-102 

Partition X as 

xon 

x (2) j 



Let 



and B = 




and consider the linear combinations AX (1) and BX (2) . Find 

(a) £(X (I) ) 

(b) £(AX (1) ) 

(c) Cov(X (1) ) 

(d) Cov(AX^) 

(e) £(X< 2 ’) 

(f) £(BX< 2) ) 

(g) Cov(X< 2) ) 

(h) Cov(BX< 2 >) 

(i) Cov(X (1, ,X< 2 >) 

0) Cov(AX<’UX (2) ) 

2.33. Repeat Exercise 2.32, but with X partitioned as 





2.34. Consider the vectorsb' = [2,-1, 4.0] and d' = [-1,3, -2,1 ]. Verify the Cauchy-Schwarz 

inequality (b'd) 2 s (b'b)(d'd). 
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2.35. Using the vectors b ; = [-4, 3] and d' = [1,1], verify the extended Cauchy-Schwarz 
inequality (b'd) 2 < (b'BbXd'B^d) if 

“ - [4 1 ] 

2.36. Find the maximum and minimum values of the quadratic form 4x\ + 4x\ + 6*1*2 for 
all points x' = [*j, * 2 ] such that x'x = 1. 

2.37. With A as given in Exercise 2.6, find the maximum value of x'A x for x'x = 1. 

2.38. Find the maximum and minimum values of the ratio x'Ax/x'x for any nonzero vectors 

x' = * 3 ] if 

" 13 -4 2“ 

A = -4 13 -2 

2 -2 10 


2.39. Show that 


A B C has (i,/)th entry 2 a iA* c */ 

(rXj)(jXr)(rXv) fl\ 


Hint: BC has (€, ;)th entry 2) b ek c kj = d ej . So A(BC) has (/, ;)th element 

k=i 

S / t \ S t 

andij + a i2 d 2j + •** + fl/j d S} = 2) a U ( 2 b ek c k) ) ~ 2 2 a ie b ekCkj 

/ e=i k~i 


240. Verify (2-24): E(X + Y) = £(X) + £(Y) and £(AXB) = AF(X)B. 

Hwt X + Y has X i} + Yjj as its (i, ;)th element. Now, E(X i} + Y i} ) = E(X i} ) + E(Y i} ) 
by a univariate property of expectation, and this last quantity is the (i, y)th element of 

£(X) + £(Y). Next (see Exercise 2.39), AXB has (/,/)th entry ^ 2 a uXek b kj> and 
by the additive property of expectation, t k 

e( 2 2 ««*«**/) = 22 a ie E(X ek )b kl 

\ e k / e k 

which is the (i, ;)th element of AF(X)B. 

2.41. You are given the random vector X'= [X \, X 2> X$ y X 4 ) with mean vector 
ytx = [3, 2, -2, 0] and variance-covariance matrix 


Let 


2x 


“3 0 0 0“ 
0 3 0 0 
0 0 3 0 
_0 0 0 3 _ 



(a) Find E (AX), the mean of AX. 

(b) Find Cov (AX), the variances and covariances of AX. 

(c) Which pairs of linear combinations have zero covariances? 
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2.42. Repeat Exercise 2.41, but with 

'3 1 1 f 
13 11 
Zx 1131 
_1 1 1 3_ 
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Sample Geometry 
and Random Sampling 

3.1 Introduction 

With the vector concepts introduced in the previous chapter, we can now delve deeper 
into the geometrical interpretations of the descriptive statistics x, S „, and R; we do so in 
Section 3.2. Many of our explanations use the representation of the columns of X as p 
vectors in n dimensions. In Section 3.3 we introduce the assumption that the observa¬ 
tions constitute a random sample. Simply stated, random sampling implies that (1) mea¬ 
surements taken on different items (or trials) are unrelated to one another and (2) the 
joint distribution of all p variables remains the same for all items. Ultimately, it is this 
structure of the random sample that justifies a particular choice of distance and dictates 
the geometry for the ^-dimensional representation of the data. Furthermore, when data 
can be treated as a random sample, statistical inferences are based on a solid foundation. 

Returning to geometric interpretations in Section 3.4, we introduce a single 
number, called generalized variance , to describe variability. This generalization of 
variance is an integral part of the comparison of multivariate means. In later sec¬ 
tions we use matrix algebra to provide concise expressions for the matrix products 
and sums that allow us to calculate x and S„ directly from the data matrix X. The 
connection between x, S„, and the means and covariances for linear combinations 
of variables is also clearly delineated, using the notion of matrix products. 

3.2 The Geometry of the Sample 

A single multivariate observation is the collection of measurements on p different 
variables taken on the same item or trial. As in Chapter 1, if n observations have 
been obtained, the entire data set can be placed in an n X p array (matrix): 


*11 

*12 * 

•• *1/7 

X = * 21 

*22 • 

*2/7 

(nxp) 




*n2 * 

*n p. 
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Each row of X represents a multivariate observation. Since the entire set of 
measurements is often one particular realization of what might have been 
observed, we say that the data are a sample of size n from a p-variate 3 
“population The sample then consists of n measurements, each of which has p i 
components. 

As we have seen, the data can be plotted in two different ways. For the! 
p-dimensional scatter plot, the rows of X represent n points in p-dimensionaH 
space. We can write 



*11 

*12 • 

• *1 p 



X = 

*21 

*22 * 

• *2 p 

- 

*2 

( nxp ) 

_*nl 

**2 ■■ 

x np~ 


_ *'n- 


1st (multivariate) observation 


-nth (multivariate) observation 


The row vector x), representing the ;th observation, contains the coordinates of a 
point. 

The scatter plot of n points in p-dimensional space provides information on the 
locations and variability of the points. If the points are regarded as solid spheres, 
the sample mean vector x, given by (1-8), is the center of balance. Variability occurs 
in more than one direction, and it is quantified by the sample variance-covariance 
matrix S„. A single numerical measure of variability is provided by the determinant 
of the sample variance-covariance matrix. When p is greater than 3, this scatter 
plot representation cannot actually be graphed. Yet the consideration of the data 
as n points in p dimensions provides insights that are not readily available from 
algebraic expressions. Moreover, the concepts illustrated for p = 2 or p = 3 remain 
valid for the other cases. 


Example 3.1 (Computing the mean vector) Compute the mean vector x from the 
data matrix. 


x = 


4 1 
-1 3 
3 5 


plot the n = 3 data points in p = 2 space, and locate x on the resulting diagram. 

The first point, xi, has coordinates xi = [4,1]. Similarly, the remaining two 
points are x' 2 = [-1,3] and X3 = [3,5]. Finally, 


4-1+3 


1+3+5 


+] 
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2 


5 

• *3 

4 

- 

*2* 3 


2 

- 

1 

• X l 

J_l_ 

» » i 1 1 V 

2 -1 

1 2 3 4 5 

-2 

- 


Figure 3.1 A plot of the data 
matrix X as n = 3 points in p = 2 
space. 


Figure 3.1 shows that x is the balance point (center of gravity) of the scatter 
plot. _ 


The alternative geometrical representation is constructed by considering the 
data as p vectors in n-dimensional space. Here we take the elements of the columns 
of the data matrix to be the coordinates of the vectors. Let 


X 

{nxp) 


*11 

*21 

*12 ‘ 

*22 * 

•• *ip 
■■ x 2p 

= [yi i yz 1 • 

• i yp] 

(3-2) 

.*« 1 

*n2 ’ 

*n p ~ 





Then the coordinates of the first point yi = [x n , x 21 ,..., x nl ] are the n measure¬ 
ments on the first variable. In general, the zth point yj = [x lf -, x 2l *,..., x ni ] is 
determined by the n-tuple of all measurements on the zth variable. In this geo¬ 
metrical representation, we depict yi,..., y p as vectors rather than points, as in the 
p-dimensional scatter plot. We shall be manipulating these quantities shortly using 
the algebra of vectors discussed in Chapter 2. 


Example 3.2 (Data as p vectors in n dimensions) Plot the following data as p = 2 
vectors in n = 3 space: 


x = 


4 

-1 

3 


1 

3 

5 


3 
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3 



Figure 3.2 A plot of the data 
matrix X as p = 2 vectors in 
n ~ 3 space. 


Here yj = [4, -1, 3] and y 2 ~ [1, 3, 5]. These vectors are shown in Figure 3.2. ■ 


Many of the algebraic expressions we shall encounter in multivariate analysis 
can be related to the geometrical notions of length, angle, and volume. This is im¬ 
portant because geometrical representations ordinarily facilitate understanding and 
lead to further insights. 

Unfortunately, we are limited to visualizing objects in three dimensions, and 
consequently, the n-dimensional representation of the data matrix X may not seem 
like a particularly useful device for n > 3. It turns out, however, that geometrical 
relationships and the associated statistical concepts depicted for any three vectors 
remain valid regardless of their dimension. This follows because three vectors, even if 
n dimensional, can span no more than a three-dimensional space, just as two vectors 
with any number of components must lie in a plane. By selecting an appropriate 
three-dimensional perspective—that is, a portion of the n-dimensional space con¬ 
taining the three vectors of interest—a view is obtained that preserves both lengths 
and angles. Thus, it is possible, with the right choice of axes, to illustrate certain alge¬ 
braic statistical concepts in terms of only two or three vectors of any dimension n. 
Since the specific choice of axes is not relevant to the geometry, we shall always 
label the coordinate axes 1,2, and 3. 

It is possible to give a geometrical interpretation of the process of finding a sam¬ 
ple mean. We start by defining the n X 1 vector Xi = [1,1,...,1]. (To simplify the 
notation, the subscript n will be dropped when the dimension of the vector X is 
clear from the context.) The vector 1 forms equal angles with each of the n 
coordinate axes, so the vector (l/ Vn)l has unit length in the equal-angle direction. 
Consider the vector y* = [jc Wi x lh ..., x ni ]. The projection of y ; on the unit vector 
(l/Vn)l is, by (2-8), 



x u + x 2i + ••• + x„ 


Vn I Vii 


1 — Jt,l 


(3-3) 
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Further, for each y,, we have the decomposition 



where jc,1 is perpendicular to y, — The deviation, or mean corrected, vector is 


d f = y, - jc r l = 


x u - Xi 

X 2 i - Xi 


\_X n j XiJ 


(3-4) 


The elements of d, are the deviations of the measurements on the *th variable from 
their sample mean. Decomposition of the y, vectors into mean components and 
deviation from the mean components is shown in Figure 3.3 for p = 3 and n = 3. 



Figure 3.3 The decomposition 
of y, into a mean component 
Xjl and a deviation component 
d,» = y i ~ x,l, i = 1, 2, 3. 


Example 3.3 (Decomposing a vector into its mean and deviation components) Let 

us carry out the decomposition of y ■ into x{l and d, = y t - x,l, i — 1,2, for the data 
given in Example 3.2: 


X = 


4 1 
-1 3 
3 5 


Here, x\ = (4 — 1 + 3)/3 = 2 and x 2 ~ (1 + 3 + 5)/3 = 3, so 


Xjl = 2 

~ 1 ~ 

1 


‘2" 

2 

II 

IH 

"r 

1 

_ 

1 

U> U) 
_ 1 


_1_ 


2_ 


1 


|3j 


That is, the sample mean jc, = (x u + x 2i + + x ni )/n = y'jl/n corresponds to the 

multiple of 1 required to give the projection of y, onto the line determined by 1. 
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Consequently, 


4 2 2 

di = yi - ill = -1 - 2 = -3 

3 2 1 


1 3 -2 

d 2 = y2 “ *2* = 3 - 3 = 0 

5 3 2 


We note that x { l and di = yi - *il are perpendicular,because 


frlKyi ~ *il) = R 2 2] -3 =4-6 + 2-0 

1 _ 

A similar result holds for x 2 l and d 2 = y 2 ~ x 2 l. The decomposition is 


4 2 2 

y 1= -1 = 2 + -3 

3 J [_2 J [_ 1 

'll [3l f-2' 

y 2 = 3 = 3 + 0 

5 3 2 


For the time being, we are interested in the deviation (or residual) vectors 
^ = y. - x{L A plot of the deviation vectors of Figure 3.3 is given in Figure 3.4. 



Figure 3.4 The deviation 
vectors d, from Figure 3.3. 
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We have translated the deviation vectors to the origin without changing their lengths 
or orientations. 

Now consider the squared lengths of the deviation vectors. Using (2-5) and 
(3-4), we obtain 

L\ = d'd, = 2 (Xj, - S,) 2 (3-5) 

/=! 

(Length of deviation vector) 2 = sum of squared deviations 

From (1-3), we see that the squared length is proportional to the variance of 
the measurements on the zth variable. Equivalently, the length is proportional to 
the standard deviation. Longer vectors represent more variability than shorter 
vectors. 

For any two deviation vectors d, and d*, 

d/d* = J} (Xji - X;)(*/* - **) (3-6) 

/-1 

Let d ik denote the angle formed by the vectors d, and d*. From (2-6), we get 

d/d* — L^ i L^ k cos (Ojfr) 
or, using (3-5) and (3-6), we obtain 

2 (*fi ~ Xi)(x jk - x k ) = J 2 (Xji - Xi) 2 J 2 (Xjk - x k ) COS(0ik) 
y=1 v y=i v y=i 

so that [see (1-5)] 

r = * lk ■ = cos (Oik) (3-7) 

The cosine of the angle is the sample correlation coefficient. Thus, if the two 
deviation vectors have nearly the same orientation, the sample correlation will be 
close to 1. If the two vectors are nearly perpendicular, the sample correlation will 
be approximately zero. If the two vectors are oriented in nearly opposite directions, 
the sample correlation will be close to -1. 

Example 3.4 (Calculating S n and R from deviation vectors) Given the deviation vec¬ 
tors in Example 3.3, Jet us compute the sample variance-covariance matrix S„ and 
sample correlation matrix R using the geometrical concepts just introduced. 

From Example 3.3, 
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The concepts of length, angle, and projection have provided us with a geometrical 
interpretation of the sample. We summarize as follows: 

Geometrical Interpretation of the Sample 

1. The projection of a column y, of the data matrix X onto the equal angular 
vector 1 is the vector 5,1. The vector 5,1 has length V/t | 5/1. Therefore, the 
rih sample mean, x h is related to the length of the projection of y ; - on 1. 

2- The information comprising S n is obtained from the deviation vectors d, = 
y'/ ~ 5/1 = [x u - 5,, 5 2i - — x h ..., x ni — 5/]'. The square of the length of d, 
is ns u , and the (inner) product between d, and d* is ns ik } 

3. The sample correlation r ik is the cosine of the angle between d, and d*. 

.3 Random Samples and the Expected Values of 
the Sample Mean and Covariance Matrix 

In order to study the sampling variability of statistics such as x and S„ with the ulti¬ 
mate aim of making inferences, we need to make assumptions about the variables 
whose observed values constitute the data set X. 

Suppose, then, that the data have not yet been observed, but we intend to collect 
n sets of measurements on p variables. Before the measurements are made, their 
values cannot, in general, be predicted exactly. Consequently, we treat them as ran¬ 
dom variables. In this context, let the (/, fc)-th entry in the data matrix be the 
random variable X jk . Each set of measurements Xy on p variables is a random vec¬ 
tor, and we have the random matrix 



A random sample can now be defined. 

If the row vectors X\ , X 2 ,.. -, X' n in (3-8) represent independent observations 
from a common joint distribution with density function /(x) = /(xj, x 2 ,..., x p ), 
then X l5 X 2 ,..., X„ are said to form a random sample from /(x). Mathematically, 
Xj, X 2 ,..., X„ form a random sample if their joint density function is given by the 
product /(xi)/(x 2 )' • • /(x„), where /(x y ) = f(x n ,x J2 ,.... x ip ) is the density func- 
tion for the ;th row vector. 

Two points connected with the definition of random sample merit special attention: 

1. The measurements of the p variables in a single trial, such as X; = 
[Xji , X j2 ,..., X jp \, will usually be correlated. Indeed, we expect this to be the 
case. The measurements from different trials must, however, be independent. 

1 The square of the length and the inner product are {n — 1)j m and (n — respectively, when 
the divisor n — 1 is used in the definitions of the sample variance and covariance. 
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2. The independence of measurements from trial to trial may not hold when the 
variables are likely to drift over time, as with sets of p stock prices or p eco¬ 
nomic indicators. Violations of the tentative assumption of independence can 
have a serious impact on the quality of statistical inferences. 

The following examples illustrate these remarks. 

Example 3.5 (Selecting a random sample) As a preliminary step in designing a 
permit system for utilizing a wilderness canoe area without overcrowding, a natural- 
resource manager took a survey of users.The total wilderness area was divided into 
subregions, and respondents were asked to give information on the regions visited, 
lengths of stay, and other variables. 

The method followed was to select persons randomly (perhaps using a random 
number table) from all those who entered the wilderness area during a particular 
week. All persons were equally likely to be in the sample, so the more popular 
entrances were represented by larger proportions of canoeists. 

Here one would expect the sample observations to conform closely to the crite¬ 
rion for a random sample from the population of users or potential users. On the 
other hand, if one of the samplers had waited at a campsite far in the interior of the 
area and interviewed only canoeists who reached that spot, successive measurements 
would not be independent. For instance, lengths of stay in the wilderness area for dif¬ 
ferent canoeists from this group would all tend to be large. a 

Example 3.6 (A nonrandom sample) Because of concerns with future solid-waste 
disposal, an ongoing study concerns the gross weight of municipal solid waste gen¬ 
erated per year in the United States (Environmental Protection Agency). Estimated 
amounts attributed to x x = paper and paperboard waste and x 2 — plastic waste, in 
millions of tons, are given for selected years in Table 3.1. Should these measure¬ 
ments on X' = [X\, X 2 ] be treated as a random sample of size n ~ 7? No! In fact, 
except for a slight but fortunate downturn in paper and paperboard waste in 2003, 
both variables are increasing over time. 



As we have argued heuristically in Chapter 1, the notion of statistical indepen¬ 
dence has important implications for measuring distance. Euclidean distance appears 
appropriate if the components of a vector are independent and have the same vari¬ 
ances. Suppose we consider the location of the £th column Y* = [ X\ k , X 2k ,..., X nk ) 
of X, regarded as a point in n dimensions. The location of this point is determined by 
the joint probability distribution f(y k ) = f(x lk , x 2kl ... , x nk ). When the measure¬ 
ments X lk , Xi k ,.. ., X nk are a random sample, f(y k ) = f(x lk , x 2kl ..., x nk ) = 
fk( x ik)fk(x 2 k) * * ‘ fk( x nk) and, consequently, each coordinate x jk contributes equally 
to the location through the identical marginal distributions /*(*/*)• 
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If the n components are not independent or the marginal distributions are not 
identical, the influence of individual measurements (coordinates) on location is 
asymmetrical. We would then be led to consider a distance function in which the 
coordinates were weighted unequally, as in the “statistical” distances or quadratic 
forms introduced in Chapters 1 and 2. 

Certain conclusions can be reached concerning the sampling distributions of X 
and S„ without making further assumptions regarding the form of the underlying 
joint distribution of the variables. In particular, we can see how X and S„ fare as point 
estimators of the corresponding population mean vector p and covariance matrix X. 


Result 3.1. Let X 2 , X 2 ,..., X„ be a random sampje from a joint distribution that 
has mean vector p and covariance matrix X. Then X is an unbiased estimator of p, 
and its covariance matrix is 


That is, 


E(X) = p 
Cov(X) = 3 2 


(population mean vector) 

population variance-covariance matrix 
divided by sample size 


For the covariance matrix S n , 


E{ S„) = ^-^2 = 2 


so [n/(n - 1)]S„ is an unbiased estimator of X, while S n is a biased estimator with 
(bias) = E(S n ) -X = -(1 /n)X. 


Proof. Now, X = (Xi + X 2 + • ■ ■ + X n )/n. The repeated use of the properties of 
expectation in (2-24) for two vectors gives 

E(X)-£(ix, + ix,.- + ix„) 

-*(;*■) + £ (;*) + 

= i£(Xj) + -£(X 2 ) + ••• + ~E(X n ) = —fi + —/!. + ■••+—n 
n n 7 nnn n 





(X - M)(X -10' = (i i (X, - ± (X, - m; 

V" y-1 / \ n e=i 

= \ i 2 (X; - /*K X < - #»)' 

n y=i r=i 
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Cov(X) = E(X - n) (X - n)' = E E *(Xy - M)(X e - m)') I 

n \y=i f=i / i 

For j * €, each entry in Zs(Xy - m)(X^ - m)' is zero because the entry is the 3 

covariance between a component of Xy and a component of X e , and these are / 

independent. [See Exercise 3.17 and (2-29).] i 

Therefore, j| 

Cov(X)=i(i£(X ; -M)(X y -M)') | 

Since X = £(Xy - m)(X ; - m)' is the common population covariance matrix for fj 
each X y , we have M 

Cov(X) = ^ ( E E(Xj - /i)(Xj - M)') = jr (X + X +-•• + X). | 


/? terms 


To obtain the expected value of S„, we first note that (X }i - X ,) ( X jk ~ X k ) is 
the (/, £)th element of (Xy - X) (Xy - X)'. The matrix representing sums of 
squares and cross products can then be written as 


2 (Xy - X) (Xy - X)' = 2 (Xy - X)Xj + % (Xj ~ X) (~X)' 


= 2 XyX; -nXX' 

7=1 

/? n 

‘ since 2 (Xy - X) = 0 and /zX' = ^ Xy. Therefore, its expected value is 

y=l /«i 

e( 2XyX' - nXX' J = 2^(XyX;) - n£(XX r ) 

Vy=i / y=i 

For any random vector V with ^(V) = Mk and Cov(V) = Xk> we have F^W') 
Xk + MkMk- (See Exercise 3.16.) Consequently, 


F(XyX') = X + mm' and F(XX') = - X + mm' 

Using these results, we obtain 

2 ^(XyX;) - nE(XX') = nX + - n^X + MM') = (« - 1)X 

and thus, since S„ = (1 /n) ^ ^ Xy Xy — nXX’j, it follows immediately that 

E(S„) = 
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n _ 

Result 3.1 shows that the (i\ &)th entry, (n — l)' 1 ^ (Xp - A^) (X Jk - X k ), of 

y=i 

[n/(n - l)]S fl is an unbiased estimator of cr ik . However, the individual sample stan¬ 
dard deviations calculated with either n or n — 1 as a divisor, are not unbiased 
estimators of the corresponding population quantities Voy]. Moreover, the correla¬ 
tion coefficients r ik are not unbiased estimators of the population quantities p ik . 
However, the bias f(V^) - Voy], or E(r ik ) - p ik , can usually be ignored if the 
sample size n is moderately large. 

Consideration of bias motivates a slightly modified definition of the sample 
variance-covariance matrix. Result 3.1 provides us with an unbiased estimator S of X: 

(Unbiased) Sample Variance-Covariance Matrix 

(3-n) 


Here S, without a subscript, has (i, A:)th entry (n - 1) 1 ^ (Xj t - A',) (X/ k ~ X k ). 

7=1 

This definition of sample covariance is commonly used in many multivariate test 
statistics. Therefore, it will replace S„ as the sample covariance matrix in most of the 
material throughout the rest of this book. 

3.4 Generalized Variance 

With a single variable, the sample variance is often used to describe the amount of 
variation in the measurements on that variable. When p variables are observed on 
each unit, the variation is described by the sample variance-covariance matrix 

f'* = ^i| (J * ~ ~ x ' )(x > k ~ * k) } 

The sample covariance matrix contains p variances and \p{p ~ 1) potentially 
different covariances. Sometimes it is desirable to assign a single numerical value for 
the variation expressed by S. One choice for a value is the determinant of S, which 
reduces to the usual sample variance of a single characteristic when p = 1. Tliis 
determinant 2 is called the generalized sample variance: 

Generalized sample variance = | S| (3-12) 

2 Definition 2A.24 defines “determinant” and indicates one method for calculating the value of a 
determinant. 


•*11 **12 **lp 

**12 **22 *** **2 p 

.**1 p **2 p ’ ’ * **p p 
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Example 3.7 (Calculating a generalized variance) Employees (j^) and profits per 
employee (*2) for the 16 largest publishing firms in the United States are shown in 
Figure 1.3. The sample covariance matrix, obtained from the data in the April 30, 
1990, Forbes magazine article, is 

T 252.04 -68.43" 

~ [_-68.43 123.67_ 

Evaluate the generalized variance. 

In this case, we compute 

| S | = (252.04) (123.67) - (-68.43) (-68.43) = 26,487 m 

The generalized sample variance provides one way of writing the information 
on all variances and covariances as a single number. Of course, when p > 1, some 
information about the sample is lost in the process. A geometrical interpretation of 
| S | will help us appreciate its strengths and weaknesses as a descriptive summary. 

Consider the area generated within the plane by two deviation vectors 
dj = y 2 — xil and d 2 = y 2 - x 2 l- Let L di be the length of d 2 and L d , the length of 
d 2 . By elementary geometry, we have the diagram 



! 

L t j| 1 



^Height = L d , sin (6) 

_cJ 



and the area of the trapezoid is | L dl sin(0) | L dr Since cos 2 (0) + sin 2 (0) = 1, we can 
express this area as 

Area = L dl L d2 V 1 - cos 2 (0) 


From (3-5) and (3-7), 


I 2 (*/i - *i) 2 = V(» - i>n 

1 


L d 2 = V 2 ( X J 2 - * 2 ) = V(n - 1)^22 

Y y-l 


cos(0) = r l2 

Therefore, 

Area = (n - 1) V*22 Vl - r\ 2 = (n - 1)VjhJ 2 2(1 ~ r ii) (3-13) 

Also, 

igi _ ^11 ‘ yi2 l = V^V^r l2 

_ s i2 s 22j\ ^Vs^rn S 22 

= *^11 ^2 2 ~ s ll s 22 r l2 ~ -^ll- y 22( 1 — ^ 2 ) 


(3-14) 
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'I X 

1 1 \ 3 



(a) (b) 

Figure 3.6 (a) “Large” generalized sample variance for p - 3. 
(b) “Small” generalized sample variance for p = 3. 


If we compare (3-14) with (3-13), we see that 

|S| = (area f/(n - 1 ) 2 

Assuming now that |S| = (n — l) -(/7 “^(volume) 2 holds for the volume gener¬ 
ated in n space by the p - 1 deviation vectors d : , d 2 , •.., d^.j, we can establish the 
following general result for p deviation vectors by induction (see [l],p. 266): 

Generalized sample variance = |S| = (n - l) - ^ volume) 2 (3-15) 

Equation (3-15) says that the generalized sample variance, for a fixed set of data, is 
proportional to the square of the volume generated by the p deviation vectors 
di = yi - 3il, d 2 = y 2 “ x 2 l,--- t & p = y p ~ x p l. Figures 3.6(a) and (b) show 
trapezoidal regions, generated by p = 3 residual vectors, corresponding to “large” 
and “small” generalized variances. 

For a fixed sample size, it is clear from the geometry that volume, or | S |, will 
increase when the length of any d, = y, - 5,1 (or Vs^-) is increased. In addition, 
volume will increase if the residual vectors of fixed length are moved until they are 
at right angles to one another, as in Figure 3.6(a). On the other hand, the volume, 
or | S [, will be small if just one of the su is small or one of the deviation vectors lies 
nearly in the (hyper) plane formed by the others, or both. In the second case, the 
trapezoid has very little height above the plane.This is the situation in Figure 3.6(b), 
where d 3 lies nearly in the plane formed by d! and d 2 . 

3 If generalized variance is defined in terms of the sample covariance matrix S„ = [(n - 1 )A*]S, then, 
using Result 2A.ll,|S fl | = |[(« - l)/n]l„S| = |[(n - l)/n]l p \\S\ - [(« - 1)/<|S|. Consequently, 
using (3-15), we can also write the following: Generalized sample variance = | S„ | = n~ p { volume) . 
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Generalized variance also has interpretations in the p-space scatter plot representa¬ 
tion of the data. The most intuitive interpretation concerns the spread of the scatter 
about the sample mean point x' = [x l , x 2 ,..., x p ]. Consider the measure of distance- 
given in the comment below (2-19), with x playing the role of the fixed point fx and S" 1 
playing the role of A. With these choices, the coordinates x' = [jq , x 2 ,..., x p ] of the 
points a constant distance c from x satisfy 

(x - x)'S-*(x - x) = c 2 (3-16) 

[When p = 1, (x — x)'S _1 (x - x) = (xi - xif/s n is the squared distance from 
to Xi in standard deviation units.] 

Equation (3-16) defines a hyper ellipsoid (an ellipse if p ~ 2) centered at x. It 
can be shown using integral calculus that the volume of this hyperellipsoid is related 
to [ S |. In particular. 

Volume of {x: (x — x)'S -1 (x - x) < c 2 } = k p \S^c p (3-17) 

or 

(Volume of ellipsoid) 2 = (constant) (generalized sample variance) 

where the constant k p is rather formidable. 4 A large volume corresponds to a large 
generalized variance. 

Although the generalized variance has some intuitively pleasing geometrical 
interpretations, it suffers from a basic weakness as a descriptive summary of the 
sample covariance matrix S, as the following example shows. 


Example 3.8 (Interpreting the generalized variance) Figure 3.7 gives three scatter 
plots with very different patterns of correlation. 

All three data sets have x' = [2,1], and the covariance matrices are 


S = 



, r = .8 S = 



,r = 0 S = 



Each covariance matrix S contains the information on the variability of the 
component variables and also the information required to calculate the correla 
tion coefficient. In this sense, S captures the orientation and size of the pattern 
of scatter. 

The eigenvalues and eigenvectors extracted from S further describe the pattern 
in the scatter plot. For 


4 5 * 


the eigenvalues satisfy 


0 = (A - 5) 2 - 4 2 
= (A-9)(A -1) 


4 For those who are curious, k p = 2?r / ' /2 /p T{pj2), where T(r) denotes the gamma function evaluated 
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In p = 2 dimensions, the choice c 2 = 5.99 will produce an ellipse th at co ntains 
approximately 95% of the observations. The vectors 3V§l#e 1 and V5.99 e 2 are 
drawn in Figure 3.8(a). Notice how the directions are the natural axes for the ellipse, 
and observe that the lengths of these scaled eigenvectors are comparable to the size 
of the pattern in each direction. 

Next, for 



the eigenvalues satisfy 


0 = (A - 3) 2 


and we arbitrarily choose the eigenvectors so that Ai = 3, e} = [1, 0] and A 2 = 3, 
e > 2 _ i]. The vectors V3 V?99 e 2 and V3 V^99 e 2 are drawn in Figure 3.8(b). 



Figure 3.8 Axes of the mean-centered 95% ellipses for the scatter plots in 
Figure 3.7. 
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Finally, for 



the eigenvalues satisfy 


0 = (A-5) 2 -(-4) 2 
= (A - 9)(A - 1) 


and we determine the eigenvalue-eigenvector pairs Aj = 9, el = [ 1/ — 1/V S] and 

A 2 = 1, = [1/V2, l/V2].The scaled eigenvectors 3V5.99 and V5.99 e 2 are 

drawn in Figure 3.8(c). 

In two dimensions, we can often sketch the axes of the mean-centered ellipse by 
eye. However, the eigenvector approach also works for high dimensions where the 
data cannot be examined visually. 

Note: Here the generalized variance | S | gives the same value, | S | = 9, for all 
three patterns. But generalized variance does not contain any information on the 
orientation of the patterns. Generalized variance is easier to interpret when the two 
or more samples (patterns) being compared have nearly the same orientations. 

Notice that our three patterns of scatter appear to cover approximately the 
same area. The ellipses that summarize the variability 


(x - x)'S *(x - x) < c 2 


do have exactly the same area [see (3-17)], since all have | S | =9. 


As Example 3.8 demonstrates, different correlation structures are not detected 
by | S |. The situation for p > 2 can be even more obscure. 

Consequently, it is often desirable to provide more than the single number | S | 
as a summary of S. From Exercise 2.12, |S| can be expressed as the product 
A^ 2 • * * A p of the eigenvalues of S. Moreover, the mean-centered ellipsoid based on 
S' 1 [see (3-16)] has axes whose lengths are proportional to the square roots of the 
A/s (see Section 2.3).These eigenvalues then provide information on the variability 
in all directions in the p-space representation of the data. It is useful, therefore, to 
report their individual values, as well as their product. We shall pursue this topic 
later when we discuss principal components. 


Situations in which the Generalized Sample Variance Is Zero 

The generalized sample variance will be zero in certain situations. A generalized 
variance of zero is indicative of extreme degeneracy, in the sense that at least one 
column of the matrix of deviations, 

xi - x' x n - xj x 12 -x 2 • • * x lp - x 

X 2 - x' _ *21 “ *1 *22 - *2 • * ■ *2 p - * 

*n ~ X'J L*«l “ *1 */i2 *2 * ’ • x np ~ x 

= X - 1 x' (3-18) 

(nXp) (nXl)(lXp) 

can be expressed as a linear combination of the other columns. As we have shown 
geometrically, this is a case where one of the deviation vectors—for instance, dj = 
[x u - x h ..., x nl - x,-]—lies in the (hyper) plane generated by di,dj_i, 
d|+i,..., d^. 
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Result 3.2. The generalized variance is zero when, and only when, at least one de¬ 
viation vector lies in the (hyper) plane formed by all linear combinations of the 
others_j s? when the columns of the matrix of deviations in (3-18) are linearly 

dependent. 

Proof. If the columns of the deviation matrix (X - lx') are linearly dependent, 
there is a linear combination of the columns such that 

0 = ^col^X - lx') + **■ + a p co\ p (X ~ lx') 

« (X - lx')a for some a ^ 0 

But then, as you may verify, (n — 1)S = (X — lx') (X - lx') and 
(n - l)Sa = (X - lx')'(X - lx')a = 0 

so the same a corresponds to a linear dependency, a x col^S) + • ■ * + a p col p (S) = 
S a = 0, in the columns of S. So, by Result 2A.9, | S | =0. 

In the other direction, if | S | =0, then there is some linear combination Sa of the 
columns of S such that Sa = 0. That is, 0 = (n - l)Sa = (X - lx')'(X - lx') a. 
Premultiplying by a' yields 

0 = a'(X - lx')'(X - I5')a = L\ x . v)m 

and for the length to equal zero, we must have (X - lx')a = 0. Thus, the columns 
of (X - 15') are linearly dependent. ■ 

Example 3.9 (A case where the generalized variance is zero) Show that | S | = 0 for 

"12 5 
X = 4 1 6 

(3X3) j_4 0 4 


and determine the degeneracy. 
Here x' = [3,1,5], so 


v “l - 3 2 - 1 5 - 5*1 |~-2 1 0“ 
X -lx' =4-3 1-16-5= 1 0 1 

[^4-3 0 — 1 4-sJ L 1 - 1 


The deviation (column) vectors are dj = [—2,1,1], d£ = [l,0, -1], and 
^ _ j-Q ^ since d 3 = ^ + 2d 2 , there is column degeneracy. (Note that there 
is row degeneracy also.) This means that one of the deviation vectors—for example, 
0 in the plane generated by the other two residual vectors. Consequently, the 
r/iree-dimensional volume is zero. This case is illustrated in figure 3.9 and may be 
verified algebraically by showing that | S | = 0. We have 



Generalized Variance 131 


Figure 3.9 A case where the 
three-dimensional volume is zero 
(|S| = 0). 

and from Definition 2A.24, 

I SI - 3 | J (-1) 2 + (-1) _ | j(-l) J +(0)~| 1 (-1) 4 
- 3 (1 - 1) + © (-i - 0) + 0 -1 - * - 0 

When large data sets are sent and received electronically, investigators are 
sometimes unpleasantly surprised to find a case of zero generalized variance, so that 
S does not have an inverse. We have encountered several such cases, with their asso¬ 
ciated difficulties, before the situation was unmasked. A singular covariance matrix 
occurs when, for instance, the data are test scores and the investigator has included 
variables that are sums of the others. For example, an algebra score and a geometry 
score could be combined to give a total math score, or class midterm and final exam 
scores summed to give total points. Once, the total weight of a number of chemicals 
was included along with that of each component. 

This common practice of creating new variables that are sums of the original 
variables and then including them in the data set has caused enough lost time that 
we emphasize the necessity of being alert to avoid these consequences. 

Example 3.10 (Creating new variables that lead to a zero generalized variance) 

Consider the data matrix 

“l 9 10“ 

4 12 16 

X = 2 10 12 

5 8 13 

, 3 U 14 _ 

where the third column is the sum of first two columns. These data could be the num¬ 
ber of successful phone solicitations per day by a part-time and a full-time employee, 
respectively, so the third column is the total number of successful solicitations per day. 

Show that the generalized variance | S | = 0, and determine the nature of the 
dependency in the data. 
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We find that the mean corrected data matrix, with entries Xj k x k , is 




X-lx' 


PS! 

mi 


The resulting covariance matrix is 

'2.5 0 2.5' 

S = 0 25 2.5 

25 25 5.0_ 

We verify that, in this case, the generalized variance 

|S| = 2.5 2 X5+0 + 0-25 3 - 2.5 3 -.0 = 0 

In general, if the three columns of the data matrix X satisfy a linear constraint 
a j Xj , + a 2 Xj 2 + a 3 x j3 = c, a constant for all j, then + a 2 x 2 + a 3 x 3 = c, so that 

a i (x n - x,) + a 2 (Xj 2 ~ h) + a 3( x j3 ~ * 3 ) = 0 

for all j . That is, 

(X - lx')a = 0 

and the columns of the mean corrected data matrix are linearly dependent.Thus, the 
inclusion of the third variable, which is linearly related to the first two, has led to the 
case of a zero generalized variance. 

^ Whenever the columns of the mean corrected data matrix are linearly dependent, 

(n - l)Sa = (X - lx')'(X - lx')a = (X - Ix')» = » 

and S a = 0 establishes the linear dependency of the columns of S. Hence, | S | — 0. 

Since S a = 0 = 0 a, we see that a is a scaled eigenvector of S associated with an 
eigenvalue of zero. This gives rise to an important diagnostic: If we are unaware of 
any extra variables that are linear combinations of the others, we can find them by 
calculating the eigenvectors of S and identifying the one associated with a zero 
eigenvalue. That is, if we were unaware of the dependency in this example, a com¬ 
puter calculation would find an eigenvalue proportional to a = [1,1, —1], since 

'2.5 0 25] [ ll |~0] 1 

Sa = 0 2.5 25 1 = 0 = 0 1 

25 25 5.oJL~U L°J L- 1 - 

The coefficients reveal that 

l(x,l - Xj) + l(x,2 - * 2 ) + (—1)(*/3 - * 3 ) = 0 forall I 

In addition, the sum of the first two variables minus the third is a constant c for all n 
units. Here the third variable is actually the sum of the first two variables, so the 
columns of the original data matrix satisfy a linear constraint with c = 0. Because 
we have the special case c = 0, the constraint establishes the fact that the columns 
of the data matrix are linearly dependent. ■ 


mi 
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Let us summarize the important equivalent conditions for a generalized vari¬ 
ance to be zero that we discussed in the preceding example. Whenever a nonzero 
vector a satisfies one of the following three conditions, it satisfies all of them: 

(1) Sa =0 (2) a'(xy - x) = 0 for all j (3) a'xy = c for all j (c = a'x) 

s ---' N -^-■' N ^^ 

a is a scaled The linear combination The linear combination of 

eigenvector of S of the mean corrected the original data, using a, 

with eigenvalue 0. data, using a, is zero. is a constant. 

We showed that if condition (3) is satisfied—that is, if the values for one variable 
can be expressed in terms of the others—then the generalized variance is zero 
because S has a zero eigenvalue. In the other direction, if condition (1) holds, 
then the eigenvector a gives coefficients for the linear dependency of the mean 
corrected data. 

In any statistical analysis, | S | = 0 means that the measurements on some vari¬ 
ables should be removed from the study as far as the mathematical computations 
are concerned. The corresponding reduced data matrix will then lead to a covari¬ 
ance matrix of full rank and a nonzero generalized variance. The question of which 
measurements to remove in degenerate cases is not easy to answer. When there is a 
choice, one should retain measurements on a (presumed) causal variable instead of 
those on a secondary characteristic. We shall return to this subject in our discussion 
of principal components. 

At this point, we settle for delineating some simple conditions for S to be of full 
rank or of reduced rank. 


Result 3.3. If n < p, that is, (sample size) < (number of variables), then | S | = 0 
for all samples. 


Proof. We must show that the rank of S is less than or equal to p and then apply 
Result 2A.9. 

For any fixed sample, the n row vectors in (3-18) sum to the zero vector. The 
existence of this linear combination means that the rank of X - lx' is less than or 
equal to n — 1, which, in turn, is less than or equal to p — 1 because n ^ p. Since 


(n - 1) S = (X - lx)'(X - lx') 

{pXp) ( p'xn ) ( nxp ) 


the fcth column of S, col*(S), can be written as a linear combination of the columns 
of (X - 15')'. In particular, 

(n — 1) col*(S) = (X — lx')' col*(X - lx') 

= (Xik - Xk) colj(X - lx')' + ••• + (x nk - x k ) col„(X - lx')' 

Since the column vectors of (X — lx')' sum to the zero vector, we can write, for 
example, colj(X — lx')' as the negative of the sum of the remaining column vectors. 
After substituting for row^X - lx')' in the preceding equation, we can express 
col*(S) as a linear combination of the at most n — 1 linearly independent row vec¬ 
tors col 2 (X - lx')',..., col rt (X — lx')'.The rank of S is therefore less than or equal 
to n - 1, which—as noted at the beginning of the proof—is less than or equal to 
p - 1, and S is singular. This implies,from Result 2A.9, that | S | = 0. ■ 
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Result 3.4. Let the p X 1 vectors x,,x 2 .x„, where x- is the ;th row of the data 

matrix X, be realizations of the independent random vectors Xi, X 2 ,• • •, X„. Then 

1. If the linear combination a'X; has positive variance for each constant vector a * 0, 
then, provided that p < n,S has full rank with probability 1 and | S | > 0. 

2. If, with probability 1, a'X ; is a constant (for example, c) for all j, then | S | = 0. 

Proof. (Part 2). If a'X ; = + a 2 X j2 + - + a p X jp = c with probability 1, 

a'Xj = c for all /, and the sample mean of this linear combination is c = £ (a,*,, 
+ a 2 x ,2 + • • • + a p x jp)! n = fl i*i + ° 2 * 2 + 1 a p x p ~ 8 S-Then 

’jtn - *1 

(X - lx')a = <ti 

.Jtni “ *i 

a'X] - a'x 

a'x„ - a'x 

indicating Unear dependence; the conclusion follows from Result 3.2. 

The proof of Part (1) is difficult and can be found in [2]. m 

Generalized Variance Determined by | R| 
and Its Geometrical Interpretation 

The generalized sample variance is unduly affected by the variability of measure¬ 
ments on a single variable. For example, suppose some s it is either large or quite 
small Then geometrically, the corresponding deviation vector d, = (y, - 5,1) will 
be very long or very short and will therefore clearly be an important factor in deter¬ 
mining volume. Consequently, it is sometimes useful to scale aU the deviation vec¬ 
tors so that they have the same length. 

Scaling the residual vectors is equivalent to replacing each ongmal observation 
x * by its standardized value (x jk - **)/V& The sample covariance matrix of the 
standardized variables is then R, the sample correlation matrix of the original vari¬ 
ables. (See Exercise 3.13.) We define 

Generalized sample variance^ , R , (3-19) 

of the standardized variables ) V 

Since the resulting vectors 

[(x,*-x*)/V^,(x 2 *-S*)/\^I.(*„* -5*)/V^] = (y* - x*l)'M^ 

all have length Vn - 1, the generalized sample variance of the standardized vari¬ 
ables will be large when these vectors are nearly perpendicular and will be small 
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when two or more of these vectors are in almost the same direction. Employing the 
argument leading to (3-7), we readily find that the cosine of the angle 0 ik between 
(y, - x ( -l)/V^ and (y* - x k l )/V^ut is th e sample correlation coefficient r ik . 
Therefore, we can make the statement that | R | is large when all the r ik are nearly 
zero and it is small when one or more of the r ik are nearly +1 or -1. 

In sum, we have the following result: Let 



be the deviation vectors of the standardized variables. The zth deviation vectors lie 
in the direction of d ; , but all have a squared length of n — 1. The volume generated 
in p-space by the deviation vectors can be related to the generalized sample vari¬ 
ance. The same steps that lead to (3-15) produce 


Generalized sample variance 
of the standardized variables 


1) p (volume) 2 


The volume generated by deviation vectors of the standardized variables is il¬ 
lustrated in Figure 3.10 for the two sets of deviation vectors graphed in Figure 3.6. 
A comparison of Figures 3.10 and 3.6 reveals that the influence of the d 2 vector 
(large variability in x 2 ) ° n the squared volume | S | is much greater than its influ¬ 
ence on the squared volume | R |. 



Figure 3.10 The volume generated by equal-length deviation vectors of 
the standardized variables. 
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The quantities | S | and | R | are connected by the relationship 

|S| -(snS22 — s pp ) |R| (3-21) 

so 

- (n ~ 1)'|S| = (n - ms n S 22 -'S pp )\Rl (3-22) 

[The proof of (3-21) is left to the reader as Exercise 3.12.] 

Interpreting (3-22) in terms of volumes, we see from (3-15) and (3-20) that the 
squared volume (n — 1)^|S| is proportional to the squared volume (n — 1)'*|R|. 
The constant of proportionality is the product of the variances, which, in turn, is 
proportional to the product of the squares of the lengths (n — 1)5/ z of the d f . 
Equation (3-21) shows, algebraically, how a change in the-measurement scale of X u 
for example, will alter the relationship between the generalized variances. Since | R | 
is based on standardized measurements, it is unaffected by the change in scale. 
However, the relative value of |S| will be changed whenever the multiplicative 
factor Si i changes. 


Example 3.11 (Illustrating the relation between |S| and | R |) Let us illustrate the 
relationship in (3-21) for the generalized variances |S| and |R| when p = 3. 
Suppose 


S 

(3X3) 


4 3 
3 9 



Then Su =4, $22 = 9, and 533 = 1. Moreover, 



Using Definition 2A.24, we obtain 

| S | = 4 2 ^~ 1)2 + 3 1 ?(~ 1)3 + 1 1 . 2 (—1)4 

= 4(9 - 4) - 3(3 - 2) + 1(6 - 9) = 14 


1 1 1 
2 l (- 1) 2 + 5 \ 


(-l) 3 +5 \ 




18 


It then follows that 

14 = |S I = 5j RI = (4)(9)(l)(i) = 14 (check) 
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Another Generalization of Variance 

We conclude-this discussion by mentioning another generalization of variance. 
Specifically, we define the total sample variance as the sum of the diagonal elements 
of the sample variance-covariance matrix S. Thus, 

Total sample variance = 5 n + 5 2 2 + * * * + s pp (3-23) 


Example 3.12 (Calculating the total sample variance) Calculate the total sample 
variance for the variance-covariance matrices S in Examples 3.7 and 3.9. 

From Example 3.7. 


and 


S = 


252.04 

-68.43 


-68.43 
123.67 _ 


Total sample variance = s n + 522 = 252.04 + 123.67 — 375.71 
From Example 3.9, 



and 

Total sample variance = 5 n + 5 2 2 + 5 3 3 = 3 + l + l = 5 


Geometrically, the total sample variance is the sum of the squared lengths of the 
p deviation vectors d! = (yj - 3c 2 l), ..., d p = (y p - x p l), divided by n - 1. The 
total sample variance criterion pays no attention to the orientation (correlation 
structure) of the residual vectors. For instance, it assigns the same values to both sets 
of residual vectors (a) and (b) in Figure 3.6. 


3.5 Sample Mean, Covariance, and Correlation 
as Matrix Operations 

We have developed geometrical representations of the data matrix X and the de¬ 
rived descriptive statistics x and S. In addition, it is possible to link algebraically the 
calculation of x and S directly to X using matrix operations. The resulting expres¬ 
sions, which depict the relation between x, S, and the full data set X concisely, are 
easily programmed on electronic computers. 
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We have it that x t = (x u ■ 1 + x 2i • 1 + • • • + x ni • l)/n = y-l/n. Therefore, 

*1 yjl *11 *12 *1 n 1 

n 

*2 j *21 *22 ••• *2 n 1 

\ — n - — . . . 

: . n : : : 


1 *p2 X pn 


x = -Xl 

n 


That is, x is calculated from the transposed data matrix by postmultiplying by the 
vector 1 and then multiplying the result by the constant 1/n. 

Next, we create an n X p matrix of means by transposing both sides of (3-24) 
and premultiplying by 1; that is, 

*1 *2 *•* X p 

lx' = -ll'X = ** * 2 ~ X P (3-25) 


L*i x 2 x pJ 

Subtracting this result from X produces the n X p matrix of deviations (residuals) 

*11 “ *1 *12 “ *2 *1 p ~ x p 

X --ll'X = X21 7 Xl X22 7 x2 Xlp ~ Xp (3-26) 

n : : \ : 

_*«1 “ *1 x n2 *2 ”■ X np X P— 

Now, the matrix (n - 1)S representing sums of squares and cross products is just 
the transpose of the matrix (3-26) times the matrix itself, or 


(n - 1)S = 


*11 “ *1 *21 ~ *1 
*12 ~ *2 *22 “ *2 

*1 p x p *2 p X p 


x n 1 “ *1 
x n2 ~ *2 


*11 - *1 *12 “ *2 
*21 “ *1 *22 " *2 


\_X nl - Xj X n2 ~ * 2 ■ • ‘ 

X--irxY(x-iii'x) =xfi-iir)x 
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(i - ill') (i - -11') = I - —11' - -11' +3-1111' = I _ ill' 

V n ) \ n ) n n n 2 n 

To summarize, the matrix expressions relating x and S to the data set X are 


x-~Xl 

n 


X I--ll' X 


The result for S„ is similar, except that 1/n replaces l/(n - 1) as the first factor. 

The relations in (3-27) show clearly how matrix operations on the data matrix 
X lead to x and S. 

Once S is computed, it can be related to the sample correlation matrix R. The 
resulting expression can also be “inverted’' to relate R to S. We first define the pXp 
sample standard deviation matrix D 12 and compute its inverse, (D 1/2 ) -1 = D -1 / 2 . Let 

Vs^[ 0 0 

rU/2 _ 0 ^^22 **’ 0 Ol 


•*11 ^12 


5 lp s 2p 


V$n V 5 n ^/^22 

s lp s 2p 

. V?n VjTd V^22 


we have 


Vsii Vsl 


7p- 


r = D -1 / 2 SD~ 1/2 


r \p r 2p 
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Postmultiplying and premultiplying both sides of (3-29) by D 1/2 and noting that 
d -1/2 D 1/2 = D 1/2 D “ = 1 gives 

S=D 1 ' 2 RD 1 / j (3-30) 

That is, R can be obtained from the information in S, whereas S can be obtained from 
D 1/z and R. Equations (3-29) and (3-30) are sample analogs of (2-36) and (2-37). 

3.6 Sample Values of Linear Combinations of Variables 

We have introduced linear combinations of p variables in Section 2.6. In many multi¬ 
variate procedures, we are led naturally to consider a linear combination of the form 

c'X = c x X i + c 2 X 2 + • ■ ■ + c p X p 

whose observed value on the ;th trial is 

c'x; = CiXji + c 2 x j2 + • • * + c p x jpi i ~ I* 2,..., n (3-31) 

The n derived observations in (3-31) have 

(c'X! + c'x 2 + + c'x„) 

Sample mean =-- 

= c'(xj + x 2 + + *n)~ = c'* (3-32) 

Since (c'X; - c'x) 2 = (c'(x; - x)) 2 = c'(x; - X)(X; - x)'c, we have 

(c'X! - c'x) 2 + (c'x 2 - c'x) 2 + ••■ + (C'x„ - c'x) 2 
Sample variance - ' n - l 

c'(Xi - x)(xi - x)'c + C'(x 2 “ x)(x 2 — x)' c + • ‘ ’ + c '( x n ~ x)(x n — x) c 
=' ~ n - 1 

*(x, - x)(x, - x)' + (X 2 - x )(x 2 -x)'+ -■ + (x„-x)(x„ - x)' ~[ 

= c '-~ n - 1 J 

or 

Sample variance of c'X = c'Sc (3-33) 

Equations (3-32) and (3-33) are sample analogs of (2-43). They correspond to sub¬ 
stituting the sample quantities x and S for the “population” quantities fx and 1, 

respectively, in (2-43). 

Now consider a second linear combination 

b'X = byXi + b 2 X 2 + + b p X p 

whose observed value on the ;th trial is 

b'xy - biXji + £ 2*72 + • • * + bpXjp , j = (3-34) 
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It follows from (3-32) and (3-33) that the sample mean and variance of these 
derived observations are 

Sample mean of b'X — b'x 
Sample variance of b'X = b'Sb 

Moreover, the sample covariance computed from pairs of observations on 
b'X and c'X is 

Sample covariance 

_ (b'X! - b'x)(c'x! - c'x) + (b'x 2 - b'x)(c'x 2 - c'x) + ••• + (b'x n - b'x)(c'x n - c'x) 

n - 1 

b'(X! — x)(x! - x)'c + b'(x 2 - x)(x 2 - x)'c + + b'(x n - x)(x n — x)'c 

= — n - 1 

. T(*1 - *)(*1 - *)' + (*2 - *)(*2 -*)' + ■••+ (x„ - x)(x„ - x)' 

= b' -—- c 

n - 1 


Sample covariance of b'X and c'X = b'Sc (3-35) 

In sum, we have the following result. 

Result 3.5. The linear combinations 

b'X =b 1 X l + b 1 X 2 + -~+b p X p 
c X = C\X j + c 2 X 2 + * * * + CpXp 

have sample means, variances, and covariances that are related to x and S by 

Sample mean of b'X = b'x 
Sample meanofc'X = c'x 
Sample variance of b'X = b'Sb (3-36) 

Sample variance of c'X = c'Sc 

Sample covariance of b'X and c'X = b'Sc ■ 


Example 3.13 (Means and covariances for linear combinations) We shall consider 
two linear combinations and their derived values for the n = 3 observations given 
in Example 3.9 as 


X — x 21 x 22 X 

_*31 *32 * 

Consider the two linear combinations 


*11 

*12 

* i3 i r 

2 

5 

*21 

*22 

*23 = 4 

1 

6 

*31 

*32 

*33 J L4 

0 

4 


b'X = [2 2 -1] 


= 2X l + 2*2 - *3 
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and 

~xC 

c'X = [1 — 1 3] X 2 = Xi — X 2 + 3X^ 

The means, variances, and covariance will first be evaluated directly and then be 

evaluated by (3-36). . , . „ __ 

Observations on these linear combinations are obtained by replacing X u X 2y 
and X 3 with their observed values. For example, the n = 3 observations on b'X are 

b'Xj = 2xn + 2x 12 ~~ Xu ~ 2(1) + 2(2) — (5) = 1 

b'\2 = 2*21 + 2 x 2 2 “ x 23 ~ 2(4) + 2(1) — (6) = 4 

b'x 3 = 2 jc 31 + 2x 32 "* *33 = 2(4) + 2(0) - (4) = 4 

The sample mean and variance of these values are, respectively, 

(1 + 4 + 4) 

Sample mean =-- - 3 

(1 - 3) 2 + (4 - 3) 2 + (4 - 3) 2 _ o 
Sample variance - 3-1 J 

In a similar manner, the n = 3 observations on c'X are 

c'xi = lxn - ,lx 1 2 + 3x 13 = 1(1) - 1(2) + 3(5) = 14 

C '* 2 « 1(4) - 1(1) + 3(6) = 21 

c'x 3 = 1(4) - 1(0) + 3(4) = 16 

and 

(14 + 21 + 16) _ „ 

Sample mean =- - -i' 

(14 - 17) 2 + (21 - 17) 2 + (16 - 17) 2 _ ^ 

Sample variance = 

Moreover, the sample covariance, computed from the pairs of observations 
(b'xi, c'xi), (b'x 2 , c'x 2 ), and (b'x 3 , c x 3 ), is 

Sample covariance 

(1 - 3) (14 - 17) + (4 - 3) (21 - 17) + (4 - 3) (16 - 17) 9 

= _ 3~—T 2 

Alternatively, we use the sample mean vector x and sample covariance matrix S 
derived from the original data matrix X to calculate the sample means, variances, 
and covariances for the linear combinations. Thus, if only the descriptive statistics 
are of interest, we do not even need to calculate the observations b'x ; and c'xj. 
From Example 3.9, 

and S = 
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Consequently, using (3-36), we find that the two sample means for the derived 
observations are 


Sample mean of b'X = b'x = [2 2 -1] 1 =3 (check) 

5 


Sample mean of c'X = c'x = [1 -1 3] 1 = 17 (check) 

l_5_ 

Using (3-36), we also have 

Sample variance of b'X = b'Sb 


3 -f 0 

’I 1 I 

0 k 1 


[2 2 - 1 ] 


= [2 2 -1] -f =3 (check) 


Sample variance of c'X = c'Sc 


3 2 0 1 


[1 -1 3] 1 1 

L 0 1 1 


= [1 -1 3] -1 = 13 (check) 

5 

L 2 J 


Sample covariance of b'X and c'X = b'Sc 


3 -f 0 1 

= [2 2 - 1 ] 1 | -1 

o § iJ L 3 


= [2 2 -1] -1 =| (check) 

5 

L 2- 


As indicated, these last results check with the corresponding sample quantities 
computed directly from the observations on the linear combinations. ■ 


The sample mean and covariance relations in Result 3.5 pertain to any number 
of linear combinations. Consider the q linear combinations 

a n X x + a i2 X 2 + ■■■ + a ip X p , i = (3-37) 
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flnA'i 

^21 Xi 

+ 

+ 

a l2 X 2 

a 22 X 2 

+ 

+ * 

+ 

- + 

p^p 
a 2 pXp 

_ 

1 

a l2 ' ‘ 

a 22 " 

‘ fll p 

" a 2 P 

,a q \Xi 

+ 

~a q2 X 2 

+ • 

•• + 

a qp X p _ 


ki 

■ 

* 


Taking the ith row of A, a', to be b' and the kth row of A, «i, to be c' we see that 
‘ 5„ nc n jmnlv that the ith row of AX has sample mean a,* and the tth and 
Kro?AXhJSample covariance aJSa,. Note that aJS a, is the (,. fc)th e.e- 

mentofASA'. 

Result 3.6. The q linear combinations AX in (3-38) have sample mean vector Ax 
and sample covariance matrix ASA'. ■ 


Exercises ___ 

3 . 1 . Given the data matrix 


(a) Graph the scatter plot in p = 2 dimensions Ix>cate the sample mean on your diagram. 

(b) Sketch the n = 3-dimensional representation of the data, and plot the deviation 

vectors y, - Sil and y 2 ~ S 2 1- 

(c) Sketch the deviation vectors in (b) emanating from the origin Calculate the lengths 
(C) these vectors and the cosine of the angle between them. Relate these quantities to 

S„ and R. 

3.2. Given the data matrix 

3 4 

X = 6 ~2 
_3 

(a) Graph the scatterplot in p = 2 dimensions, and locate the sample mean on your diagram. 

(b) Sketch the n = 3-space representation of the data, and plot the deviation vectors 
y, - 3.1 and y 2 ~ x 2 l- 

(c) Sketch the deviation vectors in (b) emanating from the origin. Calculate their lengths 
and the cosine of the angle between them. Relate these quantities to S„ and R. 

3.3. Perform the decomposition of y, into 3,1 andy, - 3,1 using the first column of the data 
matrix in Example 3.9. 

3.4. Use the six observations on the variable in units of millions, from Table 1.1. 

(a) Find the projection on 1’= [1,1.1.1. Ml- 

(b) Calculate the deviation vector y, - M- Relate its length to the sample standard 
deviation. 
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(c) Graph (to scale) the triangle formed by y 1? j^l, and y x - jql. Identify the length of 
each component in your graph. 

(d) Repeat Parts a-c for the variable X 2 inThble 1.1. 

(e) Graph (to scale) the two deviation vectors y x - x il and y 2 - i 2 l. Calculate the 
value of the angle between them. 

3 . 5 . Calculate the generalized sample variance | S | for (a) the data matrix X in Exercise 3.1 
and (b) the data matrix X in Exercise 3.2. 

3 . 6 . Consider the data matrix 



(a) Calculate the matrix of deviations (residuals), X — lx'. Is this matrix of full rank? 
Explain. 

(b) Determine S and calculate the generalized sample variance |S |. Interpret the latter 
geometrically. 

(c) Using the results in (b), calculate the total sample variance. [See (3-23).] 

3 . 7 . Sketch the solid ellipsoids (x - x)'S _1 (x - x) < 1 [see (3-16)] for the three matrices 





(Note that these matrices have the same generalized variance |S |.) 


3 . 8 . Given 


fi 

0 

o" 

r i 

2 

O 

It 

V) 

1 

0 

and S — 

1 

Lo 

0 

1_ 

f 

L 2 

1 

2 



(a) Calculate the total sample variance for each S. Compare the results. 

(b) Calculate the generalized sample variance for each S, and compare the results. Com¬ 
ment on the discrepancies, if any, found between Parts a and b. 

3 . 9 , The following data matrix contains data on test scores, with x Y ~ score on first test, 
x 2 = score on second test, and x 3 = total score on the two tests: 


12 

17 

29 

18 

20 

38 

14 

16 

30 

20 

18 

38 

16 

19 

35 


(a) Obtain the mean corrected data matrix, and verify that the columns are linearly de¬ 
pendent. Specify an a' = [a l9 a 2s a 3 ] vector that establishes the linear dependence. 

(b) Obtain the sample covariance matrix S, and verify that the generalized variance is 
zero. Also, show that Sa = 0, so a can be rescaled to be an eigenvector correspond¬ 
ing to eigenvalue zero. 

(c) Verify that the third column of the data matrix is the sum of the first two columns. 
That is, show that there is linear dependence, with a x = 1 ,a 2 ~ 1, and a 3 = -1. 
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, when ^ generalized variance is zero, it is the columns of the mean corrected data 
3 . 10 . wnc that are linearly dependent, not necessarily those of the data 

matrix A c ~ A . 

matrix itself. Given the data 

'3 1 0" 

6 4 6 

4 2 2 

7 0 3 
_5 3 4_ 

, , obtain the mean corrected data matrix, and verify that the columns are linearly 
3 dependent. Specify an a' = [a,, a 2 . a,] vector that establishes the dependence. 

(b) Obtain the sample covariance matrix S, and verify that the generalized variance is 
zero. 

(c) Show that the columns of the data matrix are linearly independent in this case. 

3 .,Use the^ ‘°^ ^ ^ WhiCh 

3 |2. Showthat|S| = ($ 11 * 22 •"■ s pp)l R I- 

Hint' From Equation (3-30), S = D ,/2 RD^ Taking determinants gives |S| = 
|D 1/2 ||R||D 1/2 |. (See Result 2A.11.) Now examine |D ' z |. 

, riven a data matrix X and the resulting sample correlation matrix R, 
’ mnsider the standardized observations (x jk x k )/Vs kk , k 1,2,... , p, 
= 12 n. Show that these standardized quantities have sample covariance 

matrix R 

, 1 4 Consider the data matrix X in Exercise 3.1. We have n = 3 observations on p = 2 vari- 
ables *1 and X 2 . Form the linear combinations 


c'X = [-1 2] 


b'X = [2 3] 


{il¬ 

ia- 


-X, + 2X 2 


2X\ + 3X 2 


, a , Evaluate the sample means, variances, and covariance of b'X and c'X from first 
principles. That is, calculate the observed values of b'X and c’X, and then use the 
sample mean, variance, and covariance formulas. 

(b) Calculate the sample means, variances, and covariance of b'X and c'X using (3-36). 
Compare the results in (a) and (b). 


3.15. Repeat Exercise 3.14 using the data matrix 


1 4 3 
6 2 6 
8 3 3 
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and the linear combinations 

b'X = [111] X 2 
and 

~xC 

c'X = [12 -3] X 2 

_* 3 _ 

3.16. Let V be a vector random variable with mean vector ^(V) — ft v and covariance matrix 
E(\ - Mv)(V - fly)' = 2 V . Show that £(VV') = 2 V + MvMv- 

3.17. Show that, if X and Z are independent, then each component of X is 

(pxi) (<7X1) 

independent of each component of Z. 

Hint: P[X\ ^ jci , X 2 ^ x 2 , ■ •., X p ^ x p and Z\ ^ Z \,..., Z q ^ z q ] 

= P[X t S x u X 2 < x 2 ,...,X p s x^-PlZ-i =s z,,...,Z ? < z q ] 
by independence. Let x 2 ,...,x p and z 2 , • ■ ■, z q tend to infinity, to obtain 
P[X 1 X\ and Zj < zi] = P[X l < x x ] * P[Z X < z,] 
for all xi , Z \. So X\ and Z x are independent. Repeat for other pairs. 

3.18. Energy consumption in 2001, by state, from the major sources 

x\ - petroleum x 2 = natural gas 

*3 = hydroelectric power x 4 = nuclear electric power 

is recorded in quadrillions (10 15 ) of BTUs (Source: Statistical Abstract of the United 
States 2006). 

The resulting mean and covariance matrix are 

"0.7661 f0.856 0.635 0.173 0.096’ 

_ = 0.508 = 0.635 0.568 0.128 0.067 

* 0.438 0.173 0.127 0.171 0.039 

_0.16lJ |_ 0.096 0.067 0.039 0.043_ 

(a) Using the summary statistics, determine the sample mean and variance of a state’s 
total energy consumption for these major sources. 

(b) Determine the sample mean and variance of the excess of petroleum consumption 
over natural gas consumption. Also find the sample covariance of this variable with 
the total variable in part a. 

3.19. Using the summary statistics for the first three variables in Exercise 3.18, verify the 
relation 


ISI = (fn s 22 s 33 ) |R| 
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, Tn nort hern climates, roads must be cleared of snow quickly following a storm. One 
°' measure of storm severity is x, = its duration in hours, while the effectiveness of snow 
removal can be quantified by x 2 = the number of hours crews, men, and machine, spend 
to clear snow. Here are the results for 25 incidents in Wisconsin. 


Table 3.2 Snow Data 

*1 *2 

*1 *2 

*1 *2 

“ 12.5 13.7 

14.5 16.5 

8.0 17.4 

9.0 11.0 

19.5 23.6 

8.0 13.2 

9.0 32.1 

7.0 12.3 

7.0 11.8 

9.0 24.4 

6.5 18.2 

10.5 22.0 

10.0 32.5 

4.5 18.7 

7.0 15.8 

8.5 15.6 

6.5 12.0 

8.0 12.8 

3.5 26.1 

8.0 14.5 

17.5 42.3 

10.5 17.5 
12.0 21.8 

6.0 10.4 

13.0 25.6 


(a) Find the sample mean and variance of the difference x 2 - x, by first obtaining the 
summary statistics. 

thl Obtain the mean and variance by first obtaining the individual values x fl - x fl , 
^ f or j = i, 2, 25 and then calculating the mean and variance. Compare these values 

with those obtained in part a. 


Rpferences --—-- 

1 Anderson, T. W. An Introduction to Multivariate Statistical Analysis (3rd ed.). New York. 
John Wiley,2003. 

o Faton M. and M. Perlman.“The Non-Singularity of Generalized Sample Covariance 
Matrices.” Awiafr of Statistics, 1 (1973),710-717. 



Chapter 


The Multivariate Normal 
Distribution 


4.1 Introduction 


A generalization of the familiar bell-shaped normal density to several dimensions plays 
a fundamental role in multivariate analysis. In fact, most of the techniques encountered 
in this book are based on the assumption that the data were generated from a multi¬ 
variate normal distribution. While real data are never exactly multivariate normal, the 
normal density is often a useful approximation to the “true” population distribution. 

One advantage of the multivariate normal distribution stems from the fact that 
it is mathematically tractable and “nice” results can be obtained. This is frequently 
not the case for other data-generating distributions. Of course, mathematical attrac¬ 
tiveness per se is of little use to the practitioner. It turns out, however, that normal 
distributions are useful in practice for two reasons: First, the normal distribution 
serves as a bona fide population model in some instances; second, the sampling 
distributions of many multivariate statistics are approximately normal, regardless of 
the form of the parent population, because of a central limit effect. 

To summarize, many real-world problems fall naturally within the framework of 
normal theory. The importance of the normal distribution rests on its dual role as 
both population model for certain natural phenomena and approximate sampling 
distribution for many statistics. 


4.2 The Multivariate Normal Density and Its Properties 

The multivariate normal density is a generalization of the univariate normal density 
to p ^ 2 dimensions. Recall that the univariate normal distribution, with mean p 
and variance <r 2 , has the probability density function 

f(x) = — — oo < x < oo (4-1) 

Vzzrcr 2 
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Figure 4.1 A normal density 
with mean p and variance cr 
and selected areas under the 


p- 2a - 


l+g p + 2a 


A P.o. o. .M S rf » 

ities, and thus, for the normal random variable X, 

P(jt - a £ X « M + <r) ~ 68 

P (jt, - 2tr £ X s p + 2rr) = .95 

.. Hs.nr.te the normal density function with mean p and vari- 
ft is convemen refers to the function in (4-!) with p = 10 

2d *=I S’notation will be extended to the multivariate case later. 

The term 


^Li±y =( x - M>( ff2 ) -i ( x - m) 


\ 

, f t v,s. univariate normal density function measures the square of 
in the exponent of the un 1 deviation units This can be generalized for a 
the distance from rtofim stand • as 

p x 1 vector % of observations on several vanables 

(x - m)'2T‘(* " #*) (4 ' 3) 

,« fnR me expected value of the random vector X, and the 
The p x 1 vector K exp^ matrixofX . [See (2 _ 30) and (2-31).] We 

px p matrix X is the v positive definite, so the expression in 

shall . K u m « I... ltos*. 

(4-3) is the square of the g obtained by replacing the univariate distance 

The multivariate of (4-3) in the density function of 

in (4-2) by the muluvar g univariate normalizing constant 

(4-1). When ‘his replacement s m , make s the volume 

<*0"'V >" "TJ’SS“*d”Sto~«io» uuily to, any p.lhls is ueces- 
under the surface of probabilities are represented b, volumes 

sary because, in the mu V Qf the Xj values . It can be shoW n 

under the surface over regi y 21 v I' 1 / 2 and consequently, a p-dimensional 

(see [1]) that this constant is (2ir ^ \_ >. } has the form 

normal density for the random vector a. L p 






= 12 P We shall denote this p-dimensional normal 

Sly'w i 10 ,he dens “ ! ’ in ,te 
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Example 4-1 (Bivariate normal density) Let us evaluate the p = 2-variate normal 
density in terms of the individual parameters p l = E(X x), p 2 “ £(^ 2 )* 
cr u = Var(A r j), (t 22 = Var(X 2 ), and p 12 - o- 12 /(Vcr n V^) = Corr(A"i, A^). 
Using Result 2A.8, we find that the inverse of the covariance matrix 


r n ^12 


°~U cr 22 ~ °\i L °"l2 


^22 v 12 


Introducing the correlation coefficient p 12 by writing rr i2 = Pi 2 V^Ti ^^ 22 » we 
obtain cr u cr 22 — <7 i 2 = ct u ct 22(1 - p \ 2 ), and the squared distance becomes 

(x - /O'X'^x ~ #0 


= [*1 “ Ml >*2 “ M 2 ] 


I 47 "22(1 ~ P12) 


cr 22 ~Pl2 Vcr 11 VS22 *1 Mi 

^-p 12 Vo-n V^22 cru L x 2 “ M2 

g~22(*i - Ml ) 2 + o-n(x 2 - M2) 2 - 2pi 2 VoTi V^22(^l - Mi) (*2 ~~ M2) 
crnO- 22 (l - Pi 2 ) 



The last expression is written in terms of the standardized values (xi - MiV^^n an£ * 
(*2 ~ M2)/V^22- 

Next, since |X| = cr u o- 22 — crj 2 = - Pi 2 ), we can substitute for X” 1 

and |X| in (4-4) to get the expression for the bivariate (p = 2) normal density 
involving the individual parameters Mi> M 2 > on, ^221 and pi 2 : 


f(* 1»*2) = 


27rVn^ 1 a- 22 (l — p\ 2 ) 



The expression in (4-6) is somewhat unwieldy, and the compact general form in 
(4-4) is more informative in many ways. On the other hand, the expression in (4-6) is 
useful for discussing certain properties of the normal distribution. For example, if the 
random variables X\ and X 2 are uncorrelated, so that p 12 = 0, the joint density can 
be written as the product of two univariate normal densities each of the form of (4-1). 





152 Chapter 4 The Multivariate Normal Distribution 


That is, f(x l9 x 2 ) = f(x x )f(x 2 ) and X x and X 2 are independent. [See (2-28).] This 
result is true in general. (See Result 4.5.) 

Two bivariate distributions with cr xx = a 22 are shown in Figure 4.2. In Figure 
4.2(a), X\ and X 2 are independent (p 12 = 0). In Figure 4.2(b), p X2 - .75. Notice how 
the presence of correlation causes the probability to concentrate along a line. « 
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From the expression in (4-4) for the density of a p-dimensional normal variable, it 
should be clear that the paths of x values yielding a constant height for the density are 
ellipsoids. That is, the multivariate normal density is constant on surfaces where the 
square of the distance (x - /t)'X~ 1 (x - /a) is constant. These paths are called contours: 

Constant probability density contour = {all x such that (x - /a)'X _1 ( x - /a) = c 2 } 

= surface of an ellipsoid centered at /a 

The axes of each ellipsoid of constant density are in the direction of the eigen¬ 
vectors of X -1 , and their lengths are proportional to the reciprocals of the square 
roots of the eigenvalues of 'ST 1 . Fortunately, we can avoid the calculation of X' 1 when 
determining the axes, since these ellipsoids are also determined by the eigenvalues 
and eigenvectors of X. We state the correspondence formally for later reference. 


Result 4.1. If X is positive definite, so that X 1 exists, then 


Xe = Ae implies X *e 



e 


so (A, e) is an eigenvalue-eigenvector pair for X corresponding to the pair (1/A, e) 
for X -1 . Also, X -1 is positive definite. 


Proof. For X positive definite and e ^ 0 an eigenvector, we have 0 < e'Xe = e'(^ e ) 
= e'(Ae) = Ae'e = A. Moreover, e = X _1 (Xe) = X~H^ e )» or e = AX _1 e, and divi¬ 
sion by A > 0 gives X -1 e = (1/A)e. Thus, (1/A, e) is an eigenvalue-eigenvector pair 
for X -1 . Also, for any p X 1 x, by (2-21) 

-|(tW so 


since each term A, ^x'e,-) 2 is nonnegative. In addition, x'e,- = 0 for all i only if 

p * 

x = 0. So x ^ 0 implies that 2 (1/A t ) (x'e,-) 2 > 0, and it follows that X' 1 is 


positive definite. 


The following summarizes these concepts: 


Contours of constant density for the p-dimensional normal distribution are 
ellipsoids defined by x such the that 

(x - mVS’Hx “ #*) = c 2 (4-7) 

These ellipsoids are centered at /a and have axes dtcVX^e,, where Xe,- = A^e*- 
for i = 1,2,..., p. 

A contour of constant density for a bivariate normal distribution with 
cr 11 = o ’22 is obtained in the following example. 
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Example 4 2 (Contours of the bivariate normal density) We shall obtain the axes of 
constant probability density contours for a bivariate normal distribution when 
a n = o 22 . From (4-7), these axes are given by the eigenvalues and eigenvectors of 
X. Here | X - AI| =0 becomes 

CT n ~ A *«=(*„ -A) 2 -<7? 2 
cr 12 o-n A 

= (A - o-n - <ri2)(A - o- n + <x, 2 ) 

Consequently, the eigenvalues are A, = tr ,i + <*n and A 2 = <J\\ - cr 12 . The eigen- 
vector ej is determined from 

\*» *»>!"? 

°"11 J L^J L e 2j 

or 

(7jjC] + (T\2 e 2 ~ (°"11 + °12) e 1 

0-i2«l + tr 11^2 = fa'll + ^12)«2 

These equations imply that ej = e 2 , and after normalization, the first eigenvalue- 
eigenvector paii is 

Ai = <r u + <ri 2 , Ci = 


Similarly, A 2 = a„ - <r 12 yields the eigenvector £ = [V V2. -1/V2]. 

When the covariance <r l2 (or correlation p 12 ) is positive, Aj_ + o'n is the 
largest eigenvalue, and its associated eigenvector e’, = [1/V5,1/V2] lies along 
the 45° line through the point fi' = [Mi, M 2 ]-This is true for any positive value o 
the covariance (correlation). Since the axes of the constant-density ellipses are 
given by icVA^ and ±cV\ 2 e 2 [see (4-7)], and the eigenvectors each have 
length unity, the major axis will be associated with the largest eigenvalue For 
positively correlated normal random variables, then, the major axis of the 
constant-density ellipses will be along the 45° line through ft. (See Figure 4.3.) 





Figure 4.3 A constant-density 
contour for a bivariate normal 
distribution with <J\ \ =* (T 22 an ^ 
<r 12 > 0 (o r Pi 2 ^ 0 ). 
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When the covariance (correlation) is negative, A 2 = cr n - cr 12 will be the largest 
eigenvalue, and the major axes of the constant-density ellipses will lie along a line 
at right angles to the 45° line through /a. (These results are true only for 

^ll = cr 22 -) 

To summarize, the axes of the ellipses of constant density for a bivariate normal 
distribution with cr n = cr 22 are determined by 


±c\Za u + cr 12 



We show in Result 4.7 that the choice c 2 = *£(a), where *£(a) is the upper 
(100a)th percentile of a chi-square distribution with p degrees of freedom, leads to 
contours that contain (1 - a) X 100% of the probability. Specifically, the following 
is true for a p-dimensional normal distribution: 


The solid ellipsoid of x values satisfying 

(x - m)'2 _1 (* - m) s x\( a ) ( 4 ' 8 ) 

has probability 1 - a. 


The constant-density contours containing 50% and 90% of the probability under 
the bivariate normal surfaces in Figure 4.2 are pictured in Figure 4.4. 



Figure 4-4 The 50% and 90% contours for the bivariate normal 
distributions in Figure 4.2. 

The p-variate normal density in (4-4) has a maximum value when the squared 
distance in (4-3) is zero—that is, when x = /a. Thus, /a is the point of maximum 
density, or mode , as well as the expected value of X, or mean. The fact that /a is 
the mean of the multivariate normal distribution follows from the symmetry 
exhibited by the constant-density contours: These contours are centered, or balanced, 
at /a. 
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Additional Properties of the Multivariate 
Normal Distribution 

Certain properties of the normal distribution will be needed repeatedly in our 
explanations of statistical models and methods. These properties make it possible 
to manipulate normal distributions easily and, as we suggested in Section 4.1, are 
partly responsible for the popularity of the normal distribution. The key proper¬ 
ties, which we shall soon discuss in some mathematical detail, can be stated rather 
simply. 

The following are true for a random vector X having a multivariate normal 
distribution: 

1. Linear combinations of the components of X are normally distributed. 

2. All subsets of the components of X have a (multivariate) normal distribution. 

3. Zero covariance implies that the corresponding components are independently 
distributed. 

4. The conditional distributions of the components are (multivariate) normal. 



These statements are reproduced mathematically in the results that follow. Many 
of these results are illustrated with examples. The proofs that are included should 
help improve your understanding of matrix manipulations and also lead you 
to an appreciation for the manner in which the results successively build on 
themselves. 

Result 4.2 can be taken as a working definition of the normal distribution. With 
this in hand, the subsequent properties are almost immediate. Our partial proof of 
Result 4.2 indicates how the linear combination definition of a normal density 
relates to the multivariate density in (4-4). 


Result 4.2. If X is distributed as N p (fi, X), then any linear combination of vari¬ 
ables a'X = aiX x + a 2 X 2 + ••• + a p X p is distributed as A(a'^i, a'Xa). Also, if a'X 
is distributed as A(a'/t, a'Xa) for every a, then X must be N p {fi , X). 


Proof. The expected value and variance of a'X follow from (2-43). Proving that 
a'Xis normally distributed if X is multivariate normal is more difficult. You can find 
a proof in [1]. The second part of result 4.2 is also demonstrated in [1]. ■ 


Example 4.3 (The distribution of a linear combination of the components of a normal 
random vector) Consider the linear combination a'X of a multivariate normal ran¬ 
dom vector determined by the choice a' = [1,0,..., 0]. Since 




and it follows from Result 4.2 that X 1 is distributed as N(/i lt <r n ). More generally, 
the marginal distribution of any component X t - of X is N(fi h o-J. ■ 

The next result considers several linear combinations of a multivariate normal 
vector X. 


Result 4.3. If X is distributed as N p (fi , X), the q linear combinations 

a n X x + ••• + a lp X p 
A X = * 21^1 + + a 2p X p 

(qXp)(pXl) i 

- a q\X\ + ••* + a qP X p _ 

are distributed as A^(A M ,AXA'). Also, X + d , where d is a vector of 
. (pxi) (pxi) 

constants, is distributed as N p (p + d, X). 

Proof. The expected value £(AX) and the covariance matrix of AX follow from 
Any linear combination b'(AX) is a linear combination of X, of the 
form a'X with a = A'b. Thus, the conclusion concerning AX follows directly from 
Result 4.2. 7 

The second part of the result can be obtained by considering a'(X + d) = 
aX + (a d), where a'X is distributed as N(a'/ti,a'Xa). It is known from the 
univariate case that adding a constant a'd to the random variable a'X leaves the 
variance unchanged and translates the mean to a'fi + a'd = a '(/* + d). Since a 
was arbitrary, X + d is distributed as N p {n + d, X). m 


Example 4.4 (The distribution of two linear combinations of the components of a 
normal random vector) For X distributed as A 3 ( M , X), find the distribution of 
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By Result 4.3, the distribution of AX is multivariate normal with mean 

A,=r; 


Mi ~ M2 
M2 “ M3 


and covariance matrix 


1 -1 0 
0 1 -1_ 


0-11 

a \2 

0\3 

°T2 

O'22 

O' 23 

^ 13 

O’23 

°33_ 


r i 

o" 

-i 

1 

L ° 

-1_ 






- 

1 

o' 

°‘12 

O' 12 “ 

022 

On ~ 

" °23 

-1 

1 

0"l 3 

O' 22 - 

023 

023 ~ 

- 033_ 

_ 0 - 

-1_ 

12 

+ 0 2 2 


0\2 

+ (T 23 - 

(T 22 ~ 

o\ 

°"23 ■ 

- O’22 ~ 

~ 0\3 

022 

- 2(T 23 

+ tr 33 



Alternatively, the mean vector Aft and covariance matrix A2A' may be veri¬ 
fied by direct calculation of the means and covariances of the two random variables 
Yi = X Y - X 2 and Y 2 = X 2 - X 3 . m 

We have mentioned that all subsets of a multivariate normal random vector X 
are themselves normally distributed. We state this property formally as Result 4.4. 

Result 4.4. All subsets of X are normally distributed. If we respectively partition 
X, its mean vector /i, and its covariance matrix X as 


Xi 

x = 

(pxi) x 2 

_((/>-*) X1) 


M2 

((/>-*)x 1) 


Sn 

^12 

(<?X<?) 


^21 

^22 

((p-<?)x<?) 



then Xi is distributed as A l q (§iu 2n)- 


Proof. Set A = I j 0 1 in Result 4.3, and the conclusion follows. 

(<?xp) ! (qx(p-q ))J 

To apply Result 4.4 to an arbitrary subset of the components of X, we simply relabel 
the subset of interest as X 1 and select the corresponding component means and 
covariances as ft 3 and Xu, respectively. ■ 
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Example 4*5 (The distribution of a subset of a normal random vector) 


If X is distributed as N$(ft, X), find the distribution of 


r = w r^i v r 

M y > Ml - , ' X„ — 

1_A 4 J |_^ 4 J |j 


. We set 


O 22 O 24 
O 24 O 44 


and note that with this assignment, X,/a, and X can respectively be rearranged and 
partitioned as 


xi 


M2 


022 

024 

012 

023 

a 25 

X 4 


M 4 


O 24 

O44 

0\4 

O34 

O45 

Xi 

» M = 

Ml 

, 2 = 

0\2 

0 14 

^11 

0\ 3 

O i5 

x 3 


M 3 


023 

^34 

0\3 

0-33 

O35 

Xs_ 


_M 5 _ 


_o 2S 

O45 

0\S 

O35 

055 


Thus, from Result 4.4, for 

we have the distribution 

Afc(Mi.Zn) 


Z11 

%12 

(2x2) 

(2X3) 

^21 

^22 

(3x2) 

(3X3) 


(J 22 0*24 

0*24 0- 44 


It is clear from this example that the normal distribution for any subset can be 
expressed by simply selecting the appropriate means and covariances from the origi¬ 
nal ft and X. The formal process of relabeling and partitioning is unnecessary. ■ 

We are now in a position to state that zero correlation between normal random 
variables or sets of normal random variables is equivalent to statistical independence. 

Result 4.5. 

(a) If Xj and X 2 are independent, then Cov(Xj, X 2 ) = 0, a a, X q 2 matrix of 

(^xi) (<? 2 X1) 


If fel is N f [Al r.?n j : 


, then Xi and X 2 are independent if 


and only if X i2 = 0. 
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(c) If X! and X 2 are independent and are distributed as N qi (ji i,2n) and 


A^ 2 (/a 2 , X 22 )y respectively, then has the multivariate normal distribution 


Mi X n 

Mi ’ O' 


Proof. (See Exercise 4.14 for partial proofs based upon factoring the density 
function when 2 12 = 0 .) m 


Example 4.6 (The equivalence of zero covariance and independence for normal 
variables) Let X be N 3 (jx, X) with 


4 1 0 
2 = 1 3 0 
0 0 2 


Are Xi and X 2 independent? What about (XuX 2 ) and X 3 ? 

Since X x and X 2 have covariance <r 12 = 1, they are not independent. However, 
partitioning X and 2 as 


Xi 


r 4 1 

x 2 

, 2 = 

13 

X 3 J 


O 

O 

_1 


’X n 

^12 

(2x2) 

(2X1) 

X 21 

X 22 

.(1X2) 

(ixl). 


we see that Xi = I I and X 3 have covariance matrix 2 12 = 


. Therefore, 


(Xi, X 2 ) and X 3 are independent by Result 4.5.This implies X 3 is independent of 
Xi and also of^ 2 . ■ 

We pointed out in our discussion of the bivariate normal distribution that 
Pi 2 = 0 (zero correlation) implied independence because the joint density function 
[see (4-6)] could then be written as the product of the marginal (normal) densities of 
Xi and X 2 . This fact, which we encouraged you to verify directly, is simply a special 
case of Result 4.5 with q\ = q 2 = 1- 


Result 4 . 6 . Let X 

(Mill! a 




be distributed as N p (fx, 2) with p = 


2 = ^-—4------- L and | 2 2 2l > 0- Then the conditional distribution of Xj, given 

L.^21 ; ^22J 

that X 2 = x 2 , is normal and has 

Mean = Mi + X 12 X^ 2 (x 2 ~ M 2 ) 


and 
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Covariance = 2 n - 2 12 2 2 l2 2 i 

variable 31 ^ covariance does not de P end on the value x 2 of the conditioning 

Sy.^Take 311 ^ pr ° 0t (See Exerdse 413 ’ which Uses the densities 


A = 

{p*p) 


I s 

(qxq) I 

^12^22 

qX(p~q) 

6 ! 

I 

{p-q)Xq j 

(p~q) x (p-q) 

1 . fe 

~ Mi - X, 

J L x 2 


is jointly normal with covariance matrix AX A' given by 


X 21 ; X 22 


I 

O'’ 

= 

2 u ^12^22^21 

0 ' 


I 

0 

^ 22 _ 


S\ 1 m 2l222 l (Xz ^ and Xz “ ^2 have zero covariance, they are 

5dependent.. Moreover the quantity X x - Ml - X 12 X 2 ‘(X 2 - M2 ) has distribution 

^(0,Z„ X 12 2 22 I 2I ). Given that X 2 = x 2 ,mi + 2 12 2 2 '(x 2 - Mz ) is a constant. 

tinnaMktrih ,y 12 ^ J2 ^ 2 ~ M 2 ) and X 2 - fi 2 are independent, the condi- 

distrihution Vv* 10 1 v* -c ? 12 ^ 22 )* 2 ~ #* 2 ) is the same as the unconditional 

■”,T °‘ ;■ ;.ft * < x * Sl “ x > - <*. - *, a-i (*. -«> 

V J 1 , 12X22X2 !)’ s0 ls the random vector X, - M , - X, 2 2 ~i ( x , - M ,) 

utedTas^V f* Va ’ Ue * 2 ' Ec l uivalentl y- 8 iven that X 2 = x 2 , X 1 is distrib¬ 
uted as %( Ml + X 12 I 22 (x 2 - Mz ), X u - X 12 Xj 2 2 21 ). . 

Example 4.7 (The conditional density of a bivariate normal distribution) Tire 
Sed°Sy y giVCn that * 2 = * 2 for an X bivariate distribution, is 


/(•* 1 U 2 ) - {conditional density of X 1 given that X 2 - x 2 \ = ^ Xl ’ — 

f(* 2 ) 


N { fll+ 2 2 2 {X2 ~^ ) ’ <Tll ~^ 2 
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Here a n ~ ^ 2/^22 = <*Ti(l “ Pn)- The two terms involving x, - pj in the expo¬ 
nent of the bivariate normal density [see Equation (4-6)] become, apart from the 
multiplicative constant -1/2(1 - P 12 ). 

(*1 - Mi) 2 , (*1 ~ ~ 

- 2pn VoTI V^2 


v°n ( , 

Ml Pl2 — (-*2 M2) 


I 2 Pi 2 


(■*2 - M2) 2 


Because p 12 = cr 12 /VtrTI V^ 22 , or - tr 12 /cr 2 2 , the complete expo- 

nent is 

-1 /(*1-Ml) 2 ,, (*1 - Ml)(*2~M2) , (-t2-M2) 2 \ 

2 (Wh)l *11 " p12 *11 / 


2crn(l - P 12 ) 


TT *i 


Vtr n 

Ml - Pl2-T7= (*2 - M2) 


1 / 1 Pi 


2(1 - p 2 n ) V*ii *22 


(*2 ~ M2) 2 


2trii(l - P 12 ) 




1 (*2 ~ M 2) 2 

2 tr 22 


The constant term 27 rVcr 11 cr 22 (l P 12 ) factors as 

\Z2tt VcrTi x V<r n (l - pf 2 ) 

Dividing the joint density of X\ and X 2 by the marginal density 
f( x A = ■ * - - e -(r:-fe) 2 /2o-22 

^ 2 ' VZnV&n 

and canceling terms yields the conditional density 

„ , , fjl 1^2) 

/(xiU 2 ) - /(JC2) 

_ _ 1 e -[x,-/li-((T 1 2/<T22)(X2-/t2)] 2 /2<rii(l-pi 2 2) ) 

VSr Vo- n (l — P 12 ) 

-OO < X] < 00 

Thus, with our customary notation, the conditional distribution of X\ given that 
X 2 =X 2 is N(fll + (o'l2/ cr 22)( x 2- M 2 ). OTl(l-Pl2))- NOW, Xll-Xi 2 Z22S 2 l = 
an _ ^ 2 / 0-22 = <r n (l - P 12 ) and X 12 X 22 = 0 - 12 / 0 - 22 , agreeing with Result 4.6, 

which we obtained by an indirect method. ■ 
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For the multivariate normal situation, it is worth emphasizing the following: 

1. All conditional distributions are (multivariate) normal. 

2. The conditional mean is of the form 

Mi + Pi,q+i( x q+i “ Mg+i) + + Pi,p( x p ~ Mp) 

(4-9) 

M<7 “h fiqq+\(Xq + i ~ Mq + l) + ’** T fiq y p{Xp ~ Mp) 
where the /Ts are defined by 


£ i ,< 7 +i Puq+2 

2 , 2 X 21 = 'V 1 V 2 

_Pq,q + l Pq,q+2 

3. The conditional covariance, Si 1 - Si 2 X 22 ^ 21 , does not depend upon the value(s) 
of the conditioning variable(s). 

We conclude this section by presenting two final properties of multivariate 
normal random vectors. One has to do with the probability content of the ellipsoids 
of constant density. The other discusses the distribution of another form of linear 
combinations. 

The chi-square distribution determines the variability of the sample variance 
s 2 = su for samples from a univariate normal population. It also plays a basic role 
in the multivariate case. 



Result 4.7. Let X be distributed as N p (fi, X) with | X | >0. Then 

(a) (X - /*)'X _1 (X - m) is distributed as Xp, where x* P denotes the chi-square 
distribution with p degrees of freedom. 

(b) The N p (fi, X) distribution assigns probability 1 - a to the solid ellipsoid 
(x: (x - /*)'X _1 (x - m) - *£(«)}> where xj>( a ) denotes the upper (100a)th 
percentile of the xj> distribution. 

Proof. We know that x* P is defined as the distribution of the sum Z\ + Z\ + • • • + Z 2 p > 
where Z\ , Z 2 ,..., Z p are independent N(0, 1) random variables. Next, by the 
spectral decomposition [see Equations (2-16) and (2-21) with A = X, and see 

p 1 

Result 4.1], X 1 = 2 T“ e i e /* where Xe, = A,e ; , so X _1 e z = (1/A,)e t -. Consequently, 
1 ** 

(X-Ityx-Hx-/*) = i)(l/A,)(X-/t)'e,*:-(X-|*) = X (l/A f )(e;(X - m)) 2 = 

i= 1 i» 1 

2 [(1/VA i) eJ(X - fi)] 2 = 2 Z 2 t for instance. Now, we can write Z = A(X - /*), 

1=1 f=i 
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where 


Z = 

(#>X 1) 


z 2 


L^J 


A = 

(pxp) 


Va/' 


and X - fJL is distributed as A/ p (0,X). Therefore, by Result 4.3, Z = A(X - fx) is 
distributed as N p (0, AXA'), where 


A X A = 

(pxp)(pxp)(pxp) 



2 A / e - e - ! 

.i=l 


W 1 


V\ 2 


e 2 


—7=e 

a j j 




Va 7 ej 

"n/A 2 e 2 


.A: e p _i 


V Aj 


e i 


Va^ 


e 2 


A Cp . 


= i 


By Result 4.5, Z 1 , Z 2 , ..., Z p are independent standard normal variables, and we 
conclude that (X - /t)'X _1 (X - fx) has a ^-distribution. 

For Part b, we note that P[(X - /n)'X _1 (X - /t) < c 2 ] is the probability as¬ 
signed to the ellipsoid (X - /t) / 2~ 1 (X - fx) < c 2 by the density N p (f a,X). But 
from Part a, P[(X - /n)'X _1 (X - fx) ^ *£(<*)] = 1 - a, and Part b holds. ■ 

Remark: (Interpretation of statistical distance) Result 4.7 provides an interpreta¬ 
tion of a squared statistical distance. When X is distributed as A ! p (fi y X), 

(X - /O'X-Hx - m) 
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In terms of X 2 (see (2-22)), Z = X 2 (X - ft) has aN p (0,I p ) distribution,and 

(X - m)'2 -1 (X - M) - (X - M )'X'52 _ 1(X - m) 

= Z'Z = Z? + Z 2 2 + --- + Z| 

The squared statistical distance is calculated as if, first, the random vector X were 
transformed to p independent standard normal random variables and then the 
usual squared distance, the sum of the squares of the variables, were applied. 

Next, consider the linear combination of vector random variables 

CiXj + c 2 X 2 + ... + c n X n = [X, | X 2 i - ; X„] c (4-10) 

(pXn) (nXl) 

This linear combination differs from the linear combinations considered earlier in 
that it defines a p, x 1 vector random variable that is a linear combination of vec¬ 
tors. Previously, we discussed a single random variable that could be written as a lin¬ 
ear combination of other univariate random variables. 

Result 4-8. Let Xi, X 2 ,..., X„ be mutually independent with X ; distributed as 
N p (fXj, X). (Note that each X ; has the same covariance matrix X.) Then 

Yi “ CiXi + c 2 X 2 + + c n X n 

is distributed as C /M;> Moreover > v i and V 2 = b x X x + b 2 X 2 

+ ■ + b n X n are jointly multivariate normal with covariance matrix 


(Sc?)* (b'c)X 
(b'c)X (jjfi?)x 

n 

Consequently, Vi and V 2 are independent if b'c = ^ Cjb } = 0. 

M 

Proof. By Result 4.5(c), the np component vector 

[Xn ,..., x lp , x 2 i ,..., x 2p ,..., x np \ = [Xi,xi,...,x;] = x' 


is the squared statistical distance from X to the population mean vector /t. If one 
component has a much larger variance than another, it will contribute less to the 
squared distance. Moreover, two highly correlated random variables will contribute 
less than two variables that are nearly uncorrelated. Essentially, the use of the in¬ 
verse of the covariance matrix, (1) standardizes all of the variables and (2) elimi¬ 
nates the effects of correlation. From the proof of Result 4.7, 


is multivariate normal. In particular, X is distributed as N np (jx , X x ), where 

(npXl) 



Mi 


‘x 0 

0“ 

f = 

M2 

and X x ~ 

0 X 

0 

| ("/>* 1) 

_Mn_ 

( npXnp) 

_0 0 

x_ 


(x - m)'2 _, (x - n) = z? + z\ + ■■■ + z 2 
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The choice 


A 

(2 pXnp) 


c 2 I * ■ c„I 


where I is the p X p identity matrix, gives 


y=i 

y=i 


and AX is normal AS,A') by Result 4.3. Straightforward block multipli¬ 

cation shows that A2,A' has the first block diagonal term 


[c,X, c 2 Z,... , c„ X] [cA c 2 I,...,c„I]' = f 2 4 )2 

\/=l / 


The off-diagonal term is 


[c,X,c 2 X,...,c n 2][bil, b 2 I, ....b„I]' = (2 

\y=l / 

n 

This term is the covariance matrix for V l 7 V 2 . Consequently, when ^ Cjb f = 

i-i 


b'c = 0 , so that 


(I a) 


X = 0 , V x and V 2 are independent by Result 4.5(b). 

(p*p) 


' For sums of the type in (4-10), the property of zero correlation is equivalent to 
requiring the coefficient vectors b and c to be perpendicular. 

Example 4.8 (Linear combinations of random vectors) Let Xi,X 2 ,X 3 , and X 4 be 
independent and identically distributed 3 Xl random vectors with 

T 3 ~i r 3 -i r 

/t = —1 and 2 = -1 10 

_ L 1 J L 1 0 2 - 

We first consider a linear combination a'Xj. of the three components of X 1 . This is a 
random variable with mean 

a'/t = 3ai ~ a 2 + a 3 

and variance 

a*2 a — 3a\ + a 2 + 2a 3 — 2fljfl 2 + 2difl 3 

That is, a linear combination a'X, of the components of a random vector is a single 
random variable consisting Of a sum of terms that are each a constant times a variable. 
This is very different from a linear combination of random vectors, say, 

CiXi + c 2 X 2 + c 3 X 3 + C 4 X 4 
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which is itself a random vector. Here each term in the sum is a constant times a 
random vector. 

Now consider two linear combinations of random vectors 

1X! + 4 X 2 + 1 X 3 + i X 4 
2 1 2 2 2 3 2 4 

and 

Xi + X 2 + X 3 - 3 X 4 

Find the mean vector and covariance matrix for each linear combination of vectors 
and also the covariance between them. 

By Result 4.8 with Ci = c 2 = c 3 = c 4 = 1/2, the first linear combination has 
mean vector 

6 

(Ci + c 2 + c 3 + c 4 )/t = 2/t = -2 

2 

and covariance matrix 

3 -1 r 

(ci + c\ + c 3 + c 4 )2 — 1 X 2 = — 1 1 0 

1 0 2 _ 

For the second linear combination of random vectors, we apply Result 4.8 with 
b\ = t >2 = b 3 = 1 and b 4 = -3 to get mean vector 

"0 

(bi + bi + Z? 3 + h 4 )/t = 0 fi = 0 

_0 

and covariance matrix 


36 

-12 

12 

{b{ + *£ + *§ + ^)2 = 12 X 2 = -12 

12 

0 

12 

0 

24 


Finally, the covariance matrix for the two linear combinations of random vectors is 

~0 0 0 " 

(ci&i + c 2 b 2 + c 3 Z ?3 + c 4 6 4 )2 ~02 = 0 0 0 

.0 0 0 _ 


Every component of the first linear combination of random vectors has zero 
covariance with every component of the second linear combination of random vectors. 

If, in addition, each X has a trivariate normal distribution, then the two linear 
combinations have a joint six-variate normal distribution, and the two linear combi¬ 
nations of vectors are independent. ■ 



168 Chapter 4 The Multivariate Normal Distribution 

4.3 Sampling from a Multivariate Normal Distribution 
and Maximum Likelihood Estimation 

We discussed sampling and selecting random samples briefly ui Chapter 3. In this • 
section, we shall-be concerned with samples from ajnultivariate normal popula¬ 
tion—in particular, with the sampling distribution of X and S. 


The Multivariate Normal Likelihood 

Let us assume that the p X 1 vectors X„X 2 ,...,X n represent a random sample 
from a multivariate normal population with mean vector p and covariance matrix 
X Since X, X, X„ are mutually independent and each has distribution 
Np ( M ,X), the joint densfry function of all the observations is the product of the 
marginal normal densities: 


Joint density 


-(x r v.yi-'{x r »)/7 


of X 2 , X 2 ,...,X n J fJi 1(277)^1X1 1/2 

1 _L- e -i 

(2w) np/2 |2r /2 ,_ 


When the numerical values of the observations become available, they may be sub¬ 
stituted for the x, in Equation (4-ll).1he resulting expression, now considered as a func¬ 
tion of M and X for the fixed set of observations x„ x 2 ,..., x n , is called the likelihood. 

Many good statistical procedures employ values for the population parameters 
that “best” explain the observed data. One meaning of best is to select theparame- 
ter values that maximize the joint density evaluated at the observations. This tech¬ 
nique is called maximum likelihood estimation , and the maximizing parameter 
values are called maximum likelihood estimates. 

At this point, we shall consider maximum likelihood estimation of the parame¬ 
ters u and X for a multivariate normal population. To do so, we take the observa¬ 
tions xi x 2 .. x„ as fixed and consider the joint density of Equation (4-11) 
evaluated ai these values. The result is the likelihood function. In order to simplify 
matters we rewrite the likelihood function in another form. We shall need some ad¬ 
ditional properties for the trace of a square matrix. (The trace of a matrix is the sum 
of its diagonal elements, and the properties of the trace are discussed in Definition 

2A.28 and Result 2A.12.) 


Result 4.9. Let A be a k X k symmetric matrix and x be a k X 1 vector. Then 

(a) x'Ax = tr(x'Ax) = tr(Axx') 

(b) tr(A) = 2 A, , where the A, are the eigenvalues of A. 

1 = 1 

Proof. For Part a, we note that x'Ax is a scalar, so x'Ax = tr(x'Ax). We pointed 

out in Result 2A.12 that tr(BC) = tr(CB) for any two matrices B and C of 
dimensions, m X k and k X m, respectively. This follows because BC has 2 V* as 
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its ith diagonal element, so tr(BC) “ X Similarly, the ;th diagonal 

element of CB is 2 Cy^y, so tr(CB) = 2 ( 2 c Ji b ij) = X ( 2 b ifii) = tr ( BC )* 

i-i /=i \i=i / i-i \)=i J 

Let x' be the matrix B with m = 1, and let Ax play the role of the matrix C. Then 
tr (x'(Ax)) = tr((Ax)x'), and the result follows. 

Part b is proved by using the spectral decomposition of (2-20) to write 
A = P'AP, where PP' = I and A is a diagonal matrix with entries , A 2 ,-.., A*. 
Therefore, tr(A) = tr(P'AP) = tr(APP') = tr(A) = Ai + A2 + • ■ • + A*. ■ 

Now the exponent in the joint density in (4-11) can be simplified. By Result 4.9(a), 
(*; - - m) = tr[(xy - - M)] 

= tr[X- 1 (Xy-/*)(x;-/*)'] (4-12) 

Next, 

2 (*; - M)'X _1 (*y - M) = 2 tr U x ; - M)'2 _1 (*y ~ /*)] 

;=I /-I 

= 2 tr[X"V; - m)(*/ - /*)'] 

= tr [ s_1 (x ~m)'^ (4-13) 

since the trace of a sum of matrices is equal to the sum of the traces of the matrices, 

n 

according to Result 2A.12(b). We can add and subtract x = (1/n) 2 x j ' n eac h 

term (xj - fx) in 2 ( x j ~ /*)(*/ ~ #»)' to give 
J- 1 

n 

2 (*; - * + * ~ M)(*> ~ X + X - H)' 

I -1 

= 2 (*y - *)(*/ - X)' + 2 (x - m)(x - M)' 

i -1 /“I 

= 2 (*/• “ x)(x y - x)'+ n(x -/t)(x -/t)' (4-14) 

i -1 

n n 

because the cross-product terms, 2 (x ; - - x)(x - ft)' and 2 (5 - /t)(xy - 

M /-t 

are both matrices of zeros (See Exercise 4.15.) Consequently, using Equations (4-13) 
and (4-14), we can write the joint density of a random sample from a multivariate 
normal population as 

X exp{-tr^-'(2 (x, - “ + «(S - M,(S - m)') j 2 j ( 4 ‘ 15 ) 
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Substituting the observed values x 1} x 2l ..., x n into the joint density yields the likeli¬ 
hood function. We shall denote this function by L(/x, X), to stress the fact that it is a 
function of the (unknown) population parameters /x and X. Thus, when the vectors 
xj contain the specific numbers actually observed, we have 


L(fi ,2) 

_\hd/2 i V 1 m/2 ^ 


(2-7r) n ' , / 2 |Xr /2 


112 '(,1 / 2 (4-16) 


It will be convenient in later sections of this book to express the exponent in the like¬ 
lihood function (4-16) in different ways. In particular, we shall make use of the identity 

tr (*/ “ *)(*/ ~ *)' + "(* ~ M)(* - m)') 

= trjx-^jj (xj - x)(Xj - x)'j + n tr[2 _1 (k -/*)(*- m)'] 

= tr[x-'(i (Xj - x )(xj - x)')] + n(x - - n) (4-17) 

Maximum Likelihood Estimation of /a and 2 

The next result will eventually allow us to obtain the maximum likelihood estima¬ 
tors of ft and X. 

Result 4.10. Given a p X p symmetric positive definite matrix B and a scalar 
b > 0, it follows that 

J_ -tr(X-‘B)/2 < _L nb) pb e~ b P 

121* | B| fc v 

for all positive definite X , with equality holding only for X = (1/2Z>)B. 

(/>xp) 

Proof. Let B 1/2 be the symmetric square root of B [see Equation (2-22)], 
so B^B 1 / 2 = B, = l and B’^B' 1 / 2 = B" 1 . Then tr(X‘ ! B) = 

tr[(X _1 B 1/2 )B 1/2 ] = tr[B 1/2 (X -1 B 1/2 )]. Let 17 be an eigenvalue of B 1/2 X" 1 B 1/Z . This 
matrix is positive definite because y'B^X^B^y = (B 1/2 y) , X~ 1 (® 1/2 y) > 0 if 
B 1/2 y # 0 or, equivalently, y ^ 0. Thus, the eigenvalues 17, of B 1/2 X _1 B 1/2 are positive 
by Exercise 2.17. Result 4.9(b) then gives 

tr(2 -1 B) = tr(B 1/2 2“ 1 B 1 ' 7 ) = £17,. 

1 = 1 
P 

and IB^B^I = n 7 )i by Exercise 2.12. From the properties of determinants in 
i=l 

Result 2A.11, we can write 

| b 1 / 2 X” 1 B 1 ' /2 | = |B 1/2 ||2' , ||B 1/2 | = |2-‘ IIB 1 / 2 1 B 1/2 1 
= 12” 1 11BI = t4tIB| 
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1 IB^X^B 1 / 2 ! l = \ Vi 

]T\~ |B| "W 

Combining the results for the trace and the determinant yields 


-tr [2 _1 B]/2 = 




p i p 

Vi /2 = 1 TT - n j2 

— 1 iB|4=r ,e 


But the function rfe^/ 2 has a maximum, with respect to 17, of (2 b) b e b , occurring at 
17 = 2b. The choice 77* = 2b , for each /, therefore gives 


J_ e -tr(X"B)/2 < _J_ <2b)P»e- b P 
v 1 \b 1 ¥* 1 b ' ' 


The upper bound is uniquely attained when X = (1/2Z?)B, since, for this choice, 

B^X^B 1 / 2 = B 1/2 (2Z?)B“ 1 B 1/2 = (2b) I 

(/>*/>) 


Moreover, 


tr[2 _1 B] = tr[B 1 ' ,2 2~ 1 B 1 '' 2 ] = tr[(26)I] = 2 bp 
1 | B ,/2 2“ ] B 1/2 1 | (26)11 (2 by 


Straightforward substitution for tr[2 1 B ] and 1/| 21 6 yields the bound asserted. ■ 

The maximum likelihood estimates of /x and X are those values—denoted by jx 
and X—that maximize the function L(/x, X) in (4-16). The estimates (x and X will 
depend on the observed values Xi, x 2 ,..., x„ through the summary statistics x and S. 

Result 4.11. Let X 1 ,X 2 ,.,.,X n be a random sample from a normal population 
with mean /x and covariance X. Then 

M = X and i = ^(X r X)(X r X)' = ^S 

are the maximum likelihood estimators of /x and X, respectively. Their observed 

n 

values, x and (1 fn) 2 (*y x)(x ; - ~ x)\ are called the maximum likelihood esti- 

>=i 

mates of /x and X. 


Proof. The exponent in the likelihood function [see Equation (4-16)], apart from 
the multiplicative factor is [see (4-17)] 


"(ju 

\j=i 


x )(xy “*)') + "(* - m)'2 \x - /x) 
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By Result 4.1, X -1 is positive definite, so the distance (x - /x)'X -1 (x - m) > 0 un¬ 
less /x = x. Thus, the likelihood is maximized with respect to /x at jx = x. It remains 
to maximize 

L(fl, 2) = - - - 7 T€ _tr f rl {y?i (*y-*Kxy-x)')]A 

- ’ (2ir)' , ^|2| n/2 L V 

n 

over X. By Result 4.10 with b = rc/2 and B = 2( x y ~ x)( x y “ x )» maximum 

7=i 
n 

occurs at X = (1 /n) 2 ( x / ~ x ) (x ; - — x)' ? as stated. 

7-1 

The maximum likelihood estimators are random quantities. They are obtained by 
replacing the observations Xj, x 2 ,..., x n in the expressions for jx and X with the 
corresponding random vectors, Xi , X 2 ,..., X„. ■ 

We note that the maximum likelihood estimator X is a random vector and the 
maximum likelihood estimator X is a random matrix. The maximum likelihood 
estimates are their particular values for the given data set. In addition, the maximum 
of the likelihood is 

L(/£L, 2) =-— e~ np f 1 -J— (4-18) 

(2u-) np/2 I2I"/ 2 ^ 

or,since |X| = \(n - l)/n] p |S|, 

L(jx ,X) = constant X (generalized variance)'”/ 2 (4-19) 

The generalized variance determines the “peakedness” of the likelihood function 
and, consequently, is a natural measure of variability when the parent population is 
multivariate normal. 

Maximum likelihood estimators possess an invariance property. Let 0 be the 
maximum likelihood estimator of 0 , and consider estimating the parameter h(0) y 
which is a function of 0 . Then the maximum likelihood estimate of 

h(0) is given by h(0) ^ (4-20) 

(a function of 0) (same function of 0) 

(See [1] and [15].) For example, 

1. The maximum likelihood estimator of /*'X“V is ji'X" 1 /!, where jx ~ X and 

X = ((/i - l)//i)S are the maximum likelihood estimators of /x and X, 
respectively. 

2. The maximum likelihood estimator of is Va^, where 


= - 2 -*<) 2 


is the maximum likelihood estimator of cr fl = Var (Xi). 
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Sufficient Statistics 

From expression (4-15), the joint density depends on the whole set of observations 
Xj,x 2 . ,x„ only through the sample mean x and the sum-of-squares-and-cross- 

n 

products matrix 2 ( x y ~ x)( x ; “ x )' = (n - 1)S. We express this fact by saying 
7=1 

that x and (n ~ 1)S (or S) are sufficient statistics : 

Let Xj, X 2 ,..., X„ be a random sample from a multivariate normal population 
with mean fx and covariance X. Then 

X and S are sufficient statistics (4-21) 

The importance of sufficient statistics for normal populations is that all of the 
information about fx and X in the data matrix X is contained in x and S, regardless 
of the sample size n. This generally is not true for nonnormal populations. Since 
many multivariate techniques begin with sample means and covariances, it is pru¬ 
dent to check on the adequacy of the multivariate normal assumption. (See Section 
4.6.) If the data cannot be regarded as multivariate normal, techniques that depend 
solely on x and S may be ignoring other useful sample information. 


4*4 The Sampling Distribution of X and S 

The tentative assumption that Xi, X 2 ,..., X„ constitute a random sample from a 
normal population with mean fx and covariance X completely determines the 
sampling distributions of X and S. Here we present the results on the sampling 
distributions of X and S by drawing a parallel with the familiar univariate 
conclusions. 

In the univariate case (p = 1), we know that X is normal with mean p = 
(population mean) and variance 

1 ~ population variance 

-cr =--—;- 

n sample size 

The result for the multivariate case (p > 2) is analogous in that X has a normal 
distribution with mean [x and covariance matrix (l/n)X. 

n 

For the sample variance, recall that (n - 1)5 2 = 2 ( X j ~ X) 2 is distributed as 
? 7=1 

cr times a chi-square variable having n ~ \ degrees of freedom (d.t). In turn, this 
chi-square is the distribution of a sum of squares of independent standard normal 
random variables. That is, (n — l)s 2 is distributed as cr 2 (Z\ + ■ • * + zjj-i) = {vZi) 2 
+ •**+ (crZ n -i) 2 . The individual terms crZ,- are independently distributed as 
N(0, cr 2 ). It is this latter form that is suitably generalized to the basic sampling 
distribution for the sample covariance matrix. 
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ST <4 lilt * -- 

• f the sample covariance matrix is called the Wishart 
The sampling distribution of the s F ^ sum of independent products of 

distribution, after its d,s< ' ov ® r ^’ 1 ‘ Spec if lC ally, 

multivariate normal random vectors, f , {4 . 22) 

Wm( . | S) = Wishart distribution with m d f. ^ lL) 

- distribution of 2 z / z / 

where the Z, are each 

We summarize the sampling distnbuti 

v „ x be a random sample of sizer, from a p-variate norma/ 

Let Xr, X 2 ,.. •, X n b . matrix 2. Then 

distribution with mean p and covanan 

1 . X is distributed as N ^' { ^ ranclom matr ix with n - 1 d.f. (4-23) 

2. (n - 1 )S is distributed as a Wisha 

3 . X and S are independent. 

’h t on of X cannot be used directly to make 
Because 2 is unknown, the dlstrl U ! dependen t information about 2, and the 
inferences about p. However, P*_° allows us to construct a statistic for 

distribution of S does not dependon^ ^ $ 

making inferences about 71 ,as w results {rom mu i t ivanable distribution 

For the present, we record some distribution are derived directly 

theory. The following properties ’ d ent products, Z,Z'. Proofs can be found 
front its definition as a sum o 

in [ll- 

Properties of the Wishart Distribution 

. u/ (a 12) independently of A 2 , which is distributed as 

I. If A, is distributed as W„„(Ai l *) v a 2 |S). That is, the 

W( Ja 2 |2), then A,+A 2 is distributed as W^^A, (4-24) 

degrees of freedom add ^ ^ . distrib uted as W,„(C AC I CSC). 

J. If A Is distributed as W„( )■ ^ ^ ^ ^ d icy 

Although we do not have y P ^ of some interest to see its rather 

function of the Wishart distri u ex j S t unless the sample size n is greater 

complicated form. Hie density does n ifi value at the positive definite 

than the number of variables p. wnen 
matrix A is 




w„-i(Al2) = 


. _ l 1 _—--- p 

2P( „-i)/2 ir M/>-')/ 4 i2i ( "' ,i/2 jMt"" ^ 


A positive definite 


where T (•) is the gamma funcUon. (See [1] and [11]) 
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4.5 Large-Sample Behavior of X and S 

Suppose the quantity X is determined by a large number of independent causes 
Vi, V 2 , ■. ., V„, where the random variables V { representing the causes have approxi¬ 
mately the same variability. If X is the sum 

X = V l + V 2 + -+V n 

then the central limit theorem applies, and we conclude that X has a distribution 
that is nearly normal. This is true for virtually any parent distribution of the V^’s, pro¬ 
vided that n is large enough. 

The univariate central limit theorem also tells us that the sampling distribution 
of the sample mean, X for a large sample size is nearly normal, whatever the 
form of the underlying population distribution. A similar result holds for many 
other important univariate statistics. 

It turns out that certain multivariate statistics, like X and S, have large-sample 
properties analogous to their univariate counterparts. As the sample sizejs in¬ 
creased without bound, certain regularities govern the sampling variation in X and 
S, irrespective of the form of the parent population. Therefore, the conclusions pre¬ 
sented in this section do not require multivariate normal populations. The only 
requirements are that the parent population, whatever its form, have a mean ft and 
a finite covariance X. 

Result 4.12 (Law of large numbers). Let Y x , Y 2 . Y„ be independent observa¬ 

tions from a population with mean E(Y;) - /x.Then 

y = n + >2 + -+ Y„ 
n 

converges in probability to ft as /^increases without bound. That is, for any 
prescribed accuracy e > 0 , F[-e < 7 — m<s] approaches unity as n 00 . 

Proof. See [9]. ■ 

As a direct consequence of the law of large numbers, which says that each Xi 
converges in probability to /x,, i = 1 , 2 ,..., p, 

X converges in probability to ft (4-26) 

Also, each sample covariance s ik converges in probability to a ik y i y k = 1,2and 

S (or 2 = S n ) converges in probability to X (4-27) 

Statement (4-27) follows from writing 

{n - 1)5,* = X (X„-%)[X lk ~X k ) 
y-i 

n 

= 2 ( X ji - Mi + Mi - Xi) ( X jk - p* + P* - X k ) 

= 2 (X jt - fi,)(X jk - p*) + n{Xj - Hi){X k - Hk) 

>=1 
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Letting Yj = (X„ - /x,)(Z M - fx k ), with E(Yj) = <r lk , we see that the fust term in 
s ik converges to a, k and the second term converges to zero, by applying the law of 
large numbers. 

The practical interpretation of statements (4-26) and (4-27) is that, with high 
probability, X will be close to p. and S will be close to 2 whenever the sample size is 
large. The statement concerning X is made even more precise by a multivariate 
version of the central limit theorem. 

Result 4.13 (The central limit theorem). Let X 1 ,X 2( ...,X„ be independent 
observations from any population with mean fx and finite covariance 2. Then 

Vn (X - fx) has an approximate N p (0 y 2) distribution 

for large sample sizes. Here n should also be large relative to p. 

Proof. See [1]. “ 

The approximation provided by the central limit theorem applies to dis¬ 
crete, as well as continuous, multivariate populations. Mathematically, the limit 
is exact, and the approach to normality is often fairly rapid. Moreover, from the 
results in Section 4.4, we know that X is exactly normally distributed when the 
underlying population is normal. Thus, we would expect the central limit theo¬ 
rem approximation to be quite good for moderate n when the parent population 

is nearly normal. . . 

As we have seen, when n is large, S is close to 2 with high probability. Conse¬ 
quently, replacing X by S in the approximating normal distribution for X will have a 
negligible effect on subsequent probability calculations^ 

Result 4.7 can be used to show that n (X - /i) ^ 1 (X ~~ M) has a \ 2 P distribution 

when X is distributed as nJ^jx, ^ 2^j or, equivalently, when Vn (X - /*) bas an 

M (0,2) distribution. The ^ distribution is approximately the sampling distribution 
of /i(X - f0'£ _1 (x “ ft) when X is approximately normally distributed. Replac¬ 
ing 2~ x by S' 1 does not seriously affect this approximation for n large and much 
greater than p. 

We summarize the major conclusions of this section as follows: 



4.6 


Let Xi, X 2 ,..., X„ be independent observations from a population with mean 
p and finite (nonsingular) covariance X. Then 

Vn (X - M) is approximately N p (0> X) 
and _ _ < 4 -28) 

n(X - m)'S _1 (X - n) is approximately Xp 

for n - p large. 

In the next three sections, we consider ways of verifying the assumption of nor¬ 
mality and methods for transforming'nonnormal observations into observations 
that are approximately normal. 



Assessing the Assumption of Normality 

As we have pointed out, most of the statistical techniques discussed in subsequent 
chapters assume that each vector observation X y comes from a multivariate normal 
distribution. On the other hand, in situations where the sample size is huge and the 
techniques depend solely on the behavior of X, or distances involving X of the form 
k(X - fx)' S -l (X - ft), the assumption of normality for the individual observa¬ 
tions is less crucial. But to some degree, the quality of inferences made by these 
methods depends on how closely the true parent population resembles the multi¬ 
variate normal form. It is imperative, then, that procedures exist for detecting cases 
where the data exhibit moderate to extreme departures from what is expected 
under multivariate normality. 

We want to answer this question: Do the observations Xy appear to violate the 
assumption that they came from a normal population? Based on the properties of 
normal distributions, we know that all linear combinations of normal variables are 
normal and the contours of the multivariate normal density are ellipsoids. There¬ 
fore, we address these questions: 

1. Do the marginal distributions of the elements of X appear to be normal? What 
about a few linear combinations of the components Xf! 

2. Do the scatter plots of pairs of observations on different characteristics give the 
elliptical appearance expected from normal populations? 

3. Are there any “wild” observations that should be checked for accuracy? 

It will become clear that our investigations of normality will concentrate on the 
behavior of the observations in one or two dimensions (for example, marginal dis¬ 
tributions and scatter plots). As might be expected, it has proved difficult to con¬ 
struct a “good” overall test of joint normality in more than two dimensions because 
of the large number of things that can go wrong. To some extent, we must pay a price 
for concentrating on univariate and bivariate examinations of normality: We can 
never be sure that we have not missed some feature that is revealed only in higher 
dimensions. (It is possible, for example, to construct a nonnormal bivariate distribu¬ 
tion with normal marginals. [See Exercise 4.8.]) Yet many types of nonnormality are 
often reflected in the marginal distributions and scatter plots. Moreover, for most 
practical work, one-dimensional and two-dimensional investigations are ordinarily 
sufficient. Fortunately, pathological data sets that are normal in lower dimensional 
representations, but nonnormal in higher dimensions, are not frequently encoun¬ 
tered in practice. 


Evaluating the Normality of the Univariate Marginal Distributions 

Dot diagrams for smaller n and histograms for n > 25 or so help reveal situations 
where one tail of a univariate distribution is much longer than the other. If the his¬ 
togram for a variable X t appears reasonably symmetric, we can check further by 
counting the number of observations in certain intervals. A univariate normal distri¬ 
bution assigns probability .683 to the interval (fa - Vcr^, fa + Vo^) and proba¬ 
bility .954 to the interval (fa - lVo\ h fa + 2Vcr^). Consequently, with a large 
sample size n , we expect the observed proportion p n of the observations lying in the 
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interval (x, - Vs^ h x { + Vs~-) to be about -683. Similarly, the observed proportion 
p i2 of the observations in (*; - 2V%, x, + 2 V^) should be about .954. Using the 
normal approximation to the sampling distribution ofp, (see [9]), we observe that 
either 


| Pn - -6831 > 3 ^ 

/(.683)(317) 

1,396 

1 n = 

Vn 

1 Pi 2 - -954| >3. 

/(.954)(.046) 

.628 

V »• = 

Vn 


would indicate departures from an assumed normal distribution for the ith charac¬ 
teristic. When the observed proportions are too small, parent distributions with 
thicker tails than the normal are suggested. 

Plots are always useful devices in any data analysis. Special plots called Q-Q 
plots can be used to assess the assumption of normality. These plots can be made for 
the marginal distributions of the sample observations on each variable. They are, in 
effect, plots of the sample quantile versus the quantile one would expect to observe if 
the observations actually were normally distributed. When the points lie very nearly 
along a straight line, the normality assumption remains tenable. Normality is suspect 
if the points deviate from a straight line. Moreover, the pattern of the deviations can 
provide clues about the nature of the nonnormality. Once the reasons for the non- 
normality are identified, corrective action is often possible. (See Section 4.8.) 

To simplify notation, let x l ,x 2 ,...,x„ represent n observations on any single 
characteristic X t . Let x (1) s x (2) s s x ln) represent these observations after 
they are ordered according to magnitude. For example, x {2) is the second smallest 
observation and x (n) is the largest observation. The x (i) ’s are the sample quantiles. 
When the x (j) are distinct, exactly /' observations are less than or equal to x u) . (This 
is theoretically always true when the observations are of the continuous type, which 
we usually assume.) The proportion jjn of the sample at or to the left of is often 
approximated by (j ~ j )/ n f° r analytical convenience. 1 

For a standard normal distribution, the quantiles q (j) are defined by the relation 



P[Z s 




(4-30) 


(See Table 1 in the appendix). Here p u) is the probability of getting a value less than 
or equal to q (n in a single drawing from a standard normal population. 

The idea is to look at the pairs of quantiles {q (j) ,x u) ) with the same associated 

cumulative probability (j - \)/n. If the data arise from a normal population, the 
pairs (q u) , x (y) ) will be approximately linearly related, since oq u) + pis nearly the 
expected sample quantile. 2 


‘The 5 in the numerator of (j - \)/n is a “continuity" correction. Some authors (see [5] and [10]) 

have suggested replacing [j - i)! n by (i ~ e)/( n + ?)• 

2 A better procedure is to plot (m 0) , where m w = E(z m ) is the expected value of the jlh- 
order statistic in a sample of size n from a standard normal distribution. (See [13] for further discussion.) 
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Example 4.9 (Constructing a Q-Q plot) A sample of n = 10 observations gives the 
values in the following table: 


Ordered 

observations 

*(/) 

Probability levels 
(/ - 2 )/" 

Standard normal 
quantiles q ^ 

-1.00 

.05 

-1.645 

-.10 

.15 

-1.036 

.16 

.25 

-.674 

.41 

.35 

-.385 

.62 

.45 

-.125 

.80 

.55 

.125 

1.26 

.65 

.385 

1.54 

.75 

.674 

1.71 

.85 

1.036 

2.30 

.95 

1.645 


/ .385 j 

= e~ i n dz = .65. [See (4-30).] 
oo v2tt 

Let us now construct the Q-Q plot and comment on its appearance. The Q-Q 
plot for the foregoing data, which is a plot of the ordered data x^ against the nor¬ 
mal quantiles q^, is shown in Figure 4.5.The pairs of points (q ^, lie very near¬ 
ly along a straight line, and we would not reject the notion that these data are 
normally distributed—particularly with a sample size as small as n = 10. 



*(/> 


Figure 4*5 A Q-Q plot for the 
data in Example 4.9. 


The calculations required for Q-Q plots are easily programmed for electronic 
computers. Many statistical programs available commercially are capable of produc¬ 
ing such plots. 

The steps leading to a Q-Q plot are as follows: 

1. Order the original observations to get » *( n ) an< ^ their corresponding 

probability values (l - |)/rc, (2 - |)/«,..., (n - \)/n; 

2. Calculate the standard normal quantiles q^, q^ 2 ), •.., q[ n )\ and 

3. Plot the pairs of observations (<7(i),-*(!)), (<?( 2 )> (4(»), -*(*))> anc * exam “ 

ine the “straightness” of the outcome. 
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SC- h “ *•—*- - k "°” -— 

from a normal population. 


, vamnlo . 10 / A q.q p| 0 t for radiation data) The quality-control department of a 
Example 4.10 (AQ upion idbythe federal government to monitor 

manufacturer of ovei are dosed. Observa- 

the amount of radiatio , . dosed doors of n = 42 randomly selected 

tions of the radiation emitted through cioseauo 

ovens were made. The data are listed in a e . • 



,ej," was L'L. Can..regard 

plot, pictured in Figur _ points indicated by the circled locations in 

ITS? S-SSirtV* relative to the res, o, the 

For the radiation This 

observations with li e „ q, e avera ee of the quantiles the tied observations would 
quantile is calculated using the average oi me q H 

have if they all differed slightly. 
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Figure 4.6 A Q-Q plot of 
the radiation data (door 
closed) from Example 4.10. 
(The integers in the plot 
indicate the number of 
points occupying the same 
location.) 


The straightness of the Q-Q plot can be measured by calculating the correlation co¬ 
efficient of the points in the plot.The correlation coefficient for the Q-Q plot is defined by 



and a powerful test of normality can be based on it. (See [5], [10], and [12].) Formally, 
we reject the hypothesis of normality at level of significance a if r Q falls below the 
appropriate value in Table 4.2. 
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Example 4.11 (A correlation coefficient test for normality) Let us calculate the cor¬ 
relation coefficient r Q from the Q-Q plot of Example 4.9 (see Figure 4.5) and test 
for normality. 

Using the information from Example 4.9, we have x = .770 and 
10 to 10 

X ( X U) ~ *)%') = 8 - 584 - X (■*</) “ xf = 8472 > and X 4(i> = 8/795 

y-i /==i y=i 


Since always, q = 0, 



A test of normality at the 10% level of significance is provided by referring tq ~ .994 
to the entry in Table 4.2 corresponding to n = lOanda = .10. This entry is .9351.Since 
r Q > .9351, we do not reject the hypothesis of normality. ■ 


Instead of r Qy some software packages evaluate the original statistic proposed 
by Shapiro and Wilk [12]. Its correlation form corresponds to replacing %) b y a 
function of the expected value of standard normal-order statistics and their covari¬ 
ances. We prefer r Q because it corresponds directly to the points in the normal- 
scores plot. For large sample sizes, the two statistics are nearly the same (see [13]), so 
either can be used to judge lack of fit. 

Linear combinations of more than one characteristic can be investigated. Many 
statisticians suggest plotting 



e'jXy where Sej = AjCj 

in which Aj is the largest eigenvalue of S. Here xj = [;t yl , x j2 ,. ■ ■, x jp ] is the yth 
observation on the p variables X { , X 2i ...,X p . The linear combination e p Xj corre¬ 
sponding to the smallest eigenvalue is also frequently singled out for inspection. 
(See Chapter 8 and [6] for further details.) 


Evaluating Bivariate Normality 

We would like to check on the assumption of normality for all distributions of 
2,3,..., p dimensions. However, as we have pointed out, for practical work it is usu¬ 
ally sufficient to investigate the univariate and bivariate distributions. We consid¬ 
ered univariate marginal distributions earlier. It is now of interest to examine the 
bivariate case. 

In Chapter 1, we described scatter plots for pairs of characteristics. If the obser¬ 
vations were generated from a multivariate normal distribution, each bivariate dis¬ 
tribution would be normal, and the contours of constant density would be ellipses 
The scatter plot should conform to this structure by exhibiting an overall pattern 
that is nearly elliptical. 

Moreover, by Result 4.7, the set of bivariate outcomes x such that 
(x - - m) ^xi(- 5) 
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samnl^nh^rvL’n WC S F° Uld expect rou Shly the same percentage, 50%, of 
sample observations to lie in the ellipse given by 

{all x such that (x - x)'S' , (x - x) s ^(.5)} 

nollZ « Ve re r aCed M by itS eStimate 5 and 2-1 87 estimate S" 1 . If not, the 
normality assumption is suspect. 


consTshL of fie n. f K 6 normallty) Although not a random sample, data 
consisting of the pairs of observations (x, = sales, * 2 = profits) for the 10 largest 

companies in the world are listed inExercise 1.4. These data give 




-.002930" 

.072148 


SSS* 3 ,h ' * PPe ” d "' ^ ( 5) * 1 39 1 *» “W observation 


F x t - 155.60T I" .000253 -.0029301 f x , - 155.60' 

L *2 - 14.70 J -.002930 ,072148_ x 2 - I 4.70 S L39 

contou^S first" 5 ° % C ° nt ° Ur 0therwise the observation is outside this 

In ”“ flISt Pa ' r of observations in Exercise 1.4 is [x, ,x 2 ]’ = [108.28,17.05], 


108.28 - 155.60 f .000253 -,002930l [ 108.28 - 155 60 ' 

L 17 05 - 14.70 J [--002930 .072148J 17.05 - 14.70 

= 1.61 > 1.39 

and this point falls outside the 50% contour. The remaining nine points have gener- 

%££££,*?* I 16, ,Spe" 

Mswithm ” <™ less Han U9, a proportion. .40, of the data 

tails within the 50 /„ contour. If the observations were normally distributed we 

propordontmiehTo ? °i 5 ’ ° f '5, em t0 be Wlthm this contour ' ™s difference in 
proportions might ordinarily provide evidence for rejecting the notion of bivariate 

TS) °“' Sampl ' Si " 10 “ *°° '« teaeh this conclusion. (See 

Computing the fraction of the points within a contour and subjectively comoar- 
tng wnh the theoretical probability is a nsefal, but rathe, „u]h p,SS. 
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A somewhat more forma, method for judging the joint normality of a data set is 
based on the squared generalized distances 

d) = (x; - xj'S'V, - x), ] = 1.2, ■ ■ •, n (4-32) 

v are the sample observations. The procedure we are about to de- 

Wh K e Kl nnf 2 |imi’ted to the bivariate case; it can be used for all p a 2 
scribe is not lim " DOpu i a tion is multivariate normal and both n and n - p are 
Whe " the ' P® 30 P e ach of the squared distances d\, d\,...,d\ should behave 
greater than 25or^ ^ Resu)t 4J and Equations (4 . 26) and (4-27).] 

like a chl - S ^ Ua f dj tances are n ot independent or exactly chi-square distributed, it is 
Although these ^ wer e. ^ result i ng plot is called a chi-square plot or 

h nmml for because the chi-square distribution is a special case of the more general 

gamma distribution. (See [6].) 

To construct the chi-square plot, 

L Order the squared ^ distances in (4-32) from smallest to largest as 

2 ,^-lV.)*.*. 

100() - \)/ n q uantile of the chi ' s< l uare distribution with p degrees of freedom. 
Quantiles are specified in terms of proportions, whereas percentiles are speci¬ 
fied in terms of percentages.^ are re|ated t0 the upper percentiles of a 

The quantiles q c .p\V 2)1 J n \ 2 {( ; 4 . 

.. 1 iq rpH distribution. In particular, q c ^{{) - 5 ) /n) - X P {\n ] + 2 )/ n )- 

ch.-squared d ^ # $traight , ine through the origin having slope 1. A 

The r ° ,rJed pattern suggests lack of normality. One or two points far above 

thetoeindic^ large distances, or outlying observations, that ment further 

attention. 

- a chi-square plot) Let us construct a chi-square plot of 

Example 4.13 < C °" ? iven in Example 4.12. Hie ordered distances and the 

,0 ' p ■ 2 “ d “ ’ 10 are " sled '* 

ing table: __ 


j 

d U) 

9c ' 2 ( 10 

1 

.30 

.10 

2 

.62 

.33 

3 

1.16 

.58 

4 

1.30 

.86 

5 

1.61 

1.20 

6 

1.64 

1.60 

7 

1.71 

2.10 

8 

1.79 

2.77 

9 

3.53 

3.79 

10 

4.38 

5.99 
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Figure 4.7 A chi-square plot of the ordered distances in Example 4.13. 


A graph of the pairs 2 ((/ — |)/10), d 2 ^ is shown in Figure 4.7. The points in 
Figure 4.7 are reasonably straight. Given the small sample size it is difficult to 
reject bivariate normality on the evidence in this graph. If further analysis of the 
data were required, it might be reasonable to transform them to observations 
more nearly bivariate normal. Appropriate transformations are discussed in 
Section 4.8. ■ 

In addition to inspecting univariate plots and scatter plots, we should check mul¬ 
tivariate normality by constructing a chi-squared or d 2 plot. Figure 4.8 contains d 2 



Figure 4.8 Chi-square plots for two simulated four-variate normal data sets with n = 30. 
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plots based on two computer-generated samples of 30 four-variate normal random 
vectors. As expected, the plots have a straight-line pattern, but the top two or three 

ordered squared distances are quite variable. 

The next example contains a real data set comparable to the simulated data set 
that produced the plots in Figure 4.8. 

Example 4 14 (Evaluating multivariate normality for a four-variable data set) The 

data in Table 4.3 were obtained by taking four different measures of stiffness, 
xxx and x 4 of each of n = 30 boards. The first measurement involves sending 
a shock wave down the board, the second measurement is determined while vibrat¬ 
ing the board and the last two measurements are obtained from static tests. The 
squared distances d) = (x, - - *) are also presented in the table. 



The marginal distributions appear quite normal (see Exercise 4.33), with the 

possible exception of specimen (board) 9. .... . 

To further evaluate multivariate normality, we constructed the chi-square plot 
shown in Figure 4 9 The two specimens with the largest squared distances are clear¬ 
ly removed from the straight-line pattern. Together, with the next largest point or 
two, they make the plot appear curved at the upper end. We will return to a discus¬ 
sion of this plot in Example 4.15. ■ 

We have discussed some rather simple techniques for checking the multivariate 
normality assumption. Specifically, we advocate calculating the dj , / = 1,2,..., n 
[see Equation (4-32)] and comparing the results with / quantiles. For example, 
p-variate normality is indicated if 

1. Roughly half of the d) are less than or equal to q c< p ( .50). 



2 . 


A plot of the ordered squared distances dfa s df 2 ) s • • • < d 



Qc,p 



, respectively, is nearly 


(n) versus 
a straight 


line having slope 1 and that passes through the origin. 


(See [6] for a more complete exposition of methods for assessing normality.) 

We close this section by noting that all measures of goodness of fit suffer the same 
serious drawback. When the sample size is small, only the most aberrant behavior will 
be identified as Jack of fit. On the other hand, very large samples invariably produce 
statistically significant lack of fit. Yet the departure from the specified distribution 
may be very small and technically unimportant to the inferential conclusions. 


4.7 Detecting Outliers and Cleaning Data 

Most data sets contain one or a few unusual observations that do not seem to be¬ 
long to the pattern of variability produced by the other observations. With data 
on a single characteristic, unusual observations are those that are either very 
large or very small relative to the others. The situation can be more complicated 
with multivariate data. Before we address the issue of identifying these outliers , 
we must emphasize that not all outliers are wrong numbers.They may, justifiably, 
be part of the group and may lead to a better understanding of the phenomena 
being studied. 
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Outliers are best detected visually whenever this is possible. When the number 
of observations n is large, dot plots are not feasible. When the number of character¬ 
istics p is large, the large number of scatter plots p(p - l)/2 may prevent viewing 
them all. Even so, we suggest first visually inspecting the data whenever possible. 

What should we look for? For a single random variable, the problem is one di¬ 
mensional, anchwe look for observations that are far from the others. For instance, 
the dot diagram 

• • 

•••• • ^ 

• • • «H • ••••••• ••• • • • • • (•) 

4 -- 1 -- \—*~x 

reveals a single large observation which is circled. 

In the bivariate case, the situation is more complicated. Figure 4.10 shows a 
situation with two unusual observations. 

The data point circled in the upper right corner of the figure is detached 
from the pattern, and its second coordinate is large relative to the rest of the x 2 
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Figure 4.10 Two outliers; one univariate and one bivariate. 
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measurements, as shown by the vertical dot diagram. The second outlier, also cir¬ 
cled,^ far from the elliptical pattern of the rest of the points, but, separately, each of 
its components has a typical value.This outlier cannot be detected by inspecting the 
marginal dot diagrams. 

In higher dimensions, there can be outliers that cannot be detected from the 
univariate plots or even the bivariate scatter plots. Here a large value of 
(Xj — x) f S _1 ( x y ~~ x ) w iH suggest an unusual observation, even though it cannot be 
seen visually. 


Steps for Detecting Outliers 

1. Make a dot plot for each variable. 

2. Make a scatter plot for each pair of variables. 

3. Calculate the standardized values Zjk = (•*/* “ for j - 1,2,...,n 

and each column k = 1,2,...,/?. Examine these standardized values for large 
or small values. 

4. Calculate the generalized squared distances (xy - x)'S _1 (xy - x). Examine 
these distances for unusually large values. In a chi-square plot, these would be 
the points farthest from the origin. 

In step 3, “large” must be interpreted relative to the sample size and number of 
variables. There are n X p standardized values. When n = 100 and p ~ 5, there are 
500 values. You expect 1 or 2 of these to exceed 3 or be less than -3, even if the data 
came from a multivariate distribution that is exactly normal. As a guideline, 3.5 
might be considered large for moderate sample sizes. 

In step 4, “large” is measured by an appropriate percentile of the chi-square dis¬ 
tribution with p degrees of freedom. If the sample size is n ~ 100, we would expect 
5 observations to have values of d) that exceed the upper fifth percentile of the chi- 
square distribution. A more extreme percentile must serve to determine observa¬ 
tions that do not fit the pattern of the remaining data. 

The data we presented in Table 4.3 concerning lumber have already been 
cleaned up somewhat. Similar data sets from the same study also contained data on 
x s - tensile strength. Nine observation vectors, out of the total of 112, are given as 
rows in the following table, along with their standardized values. 


*1 

^2 

*3 

x 4 

*5 


*2 

*3 

z 4 

z 5 

1631 

1528 

1452 

1559 

1602 

.06 

-.15 

.05 

.28 

-.12 

1770 

1677 

1707 

1738 

1785 

.64 

.43 

1.07 

.94 

.60 

1376 

1190 

723 

1285 

2791 

-1.01 

-1.47 

-2.87 

-.73 


1705 

1577 

1332 

1703 

l$64 

.37 

.04 

-.43 

.81 

.13 

1643 

1535 

1510 

1494 

1582 

.11 

-.12 

.28 

.04 

-.20 

1567 

1510 

1301 

1405 

1553 

-.21 

-.22 

-.56 

-.28 

-.31 

1528 

1591 

1714 

1685 

1698 

-.38 

.10 

1.10 

.75 

.26 

1803 

1826 

1748 

2746 

1764 

.78 

1.01 

1.23 


.52 

1587 

1554 

1352 

1554 

1551 

-.13 

-.05 

-.35 

.26 

-.32 
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tv,, standardized values are based on the sample mean and variance, calculated 
r mobservations.There are two extreme standardized values. Both are too large 

from all il2 observatio ^ ^ During their investigation, the researchers recorded 

wrth standardized va and f hen performed calculations that produced the 

measurements by hand m ^gb ^ rea)rds regardil)g , h e values pin- 

pointedby reachngs on^n^hidividual^variable 

- 

~ rTT^etecting outliers in the data on lumber) Table 4.4 contains the data 
Example 4.15 Detect g observations These data consist of four 

in Table 4.3, along wtfh thestand ^ ^ Qn each of n = 30 boards Recall 

different measures of ^ ^ a shock wave down the board, the second 

that ^'emenSeSned while vibrating the board, and the last two measurements 


Table 4-4 Four 


irf^suremen tsofS tiffness with Standardized \hlues_ 

x X4 Observ ation no. Z\ _*2_ 


‘■v 

4 ^ 


1889 1651 1561 1778 

2403 2048 2087 2197 

2119 1700 1815 2222 

1645 1627 1110 1533 

1976 1916 1614 1883 

1712 1712 1439 546 

1943 1685 1271 1671 

2104 1820 1717 1874 

2983 2794 2412 2581 

1745 1600 1384 1508 

1710 1591 1518 1667 

2046 1907 1627 1898 

1840 1841 1595 1741 

1867 1685 1493 1678 

1859 1649 1389 1714 

1954 2149 1180 1281 

1325 1170 1002 1176 

1419 1371 1252 1308 

1828 1634 1602 1755 

1725 1594 1313 1646 

2276 2189 1547 2111 

1899 1614 1422 1477 

1633 1513 1290 1516 

2061 1867 1646 2037 

1856 1493 1356 1533 

1727 1412 1238 1469 

2168 1896 1701 1834 

1655 1675 1414 1597 

2326 2301 2065 2234 

1490 1382 1214 1284 


-.4 -1.3 

.5 .3 

-.1 -.2 

-.2 -.8 

.2 . .7 

3.3 3.0 

-.5 -.4 

-.5 .0 

.5 -4 

.3 -3 

-.2 -.1 

-.3 -.4 


-.2 -.8 

-.6 - 1.1 

.8 .5 

-.8 -.2 

1.3 1-7 


-.4 

-.7 

.77 

.0 

-.2 

1.93 

.4 

.5 

.46 

.3 

.0 

2.70 

-.1 

-.1 

.13 

-.4 

-.0 

1.08 

-1.1 

-1.4 

dpj) 

-1.7 

-1.7 

3.50 

-.8 

-1.3 

3.99 

.3 

.1 

1.36 

-.6 

-.2 

1.46 

.1 

1.2 

9.90 

-.3 

-.8 

5.06 

-.7 

-.6 

.80 

.5 

1.0 

2.54 

-.5 

-.6 

4.58 

-.9 

-.8 

3.40 

.6 

.3 

2.38 

-.3 

-.4 

3.00 

1.8 

1.6 

6.28 

-1.0 

-1.4 

2.58 



1200 1800 2400 1500 2500 


Detecting Outliers and Cleaning Data ,191 


1500 2500 1200 1800 2400 



1500 2500 1000 1600 2200 


Figure 4.1 I Scatter plots for the lumber stiffness data with specimens 9 and 16 plotted as solid dots. 

x ik - x k 

z jk = ~r=^, k = 1,2,3,4; j = 1,2,. ..,30 
vs kk 

and the squares of the distances are dj = (xy — x) , S“ 1 (xy - x). 

The last column in Table 4.4 reveals that specimen 16 is a multivariate outlier, 
since ^(.OOS) = 14.86; yet all of the individual measurements are well within their 
respective univariate scatters. Specimen 9 also has a large d 2 value. 

The two specimens (9 and 16) with large squared distances stand out as clearly 
different from the rest of the pattern in Figure 4.9. Once these two points are 
removed, the remaining pattern conforms to the. expected straight-line relation. 
Scatter plots for the lumber stiffness measurements are given in Figure 4.11 above. 
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The solid dots in these figures correspond to specimens 9 and 16. Although the dot for 
specimen 16 stands out in all the plots, the dot for specimen 9 is “hidden” in the scat- 
ter plot of x 3 versus x 4 and nearly hidden in that ofxi versus x 3 . However, specimen 9 
is clearly identified as a multivariate outlier when all four variables are considered. 

Scientists specializing in the properties of wood conjectured that specimen 9 
was unusually clear and therefore very stiff and strong. It would also appear that 
spedmen 16 is a bit unusual, since both of its dynamic measurements are above av¬ 
erage and the two static measurements are low. Unfortunately, it was not possible to 
investigate this specimen further because the material was no longer available, m 

If outliers are identified, they should be examined for content, as was done in 
the case of the data on lumber stiffness in Example 4.15. Depending upon the 
nature of the outliers and the objectives of the investigation, outliers may be delet¬ 
ed or appropriately “weighted” in a subsequent analysis. 

Even though many statistical techniques assume normal populations, those 
based on the sample mean vectors usually will not be disturbed by a few moderate 
outliers. Hawkins [7] gives an extensive treatment of the subject of outliers. 

4.8 Transformations to Near Normality 

If normality is not a viable assumption, what is the next step? One alternative is to 
ignore the findings of a normality check and proceed as if the data were normally 
distributed. This practice is not recommended, since, in many instances, it could lead 
to incorrect conclusions. A second alternative is to make nonnormal data more 
“normal looking” by considering transformations of the data. Normal-theory analy¬ 
ses can then be carried out with the suitably transformed data. 

Transformations are nothing more than a reexpression of the data in different 
units. For example, when a histogram of positive observations exhibits a long right- 
hand tail, transforming the observations by taking their logarithms or square roots 
will often markedly improve the symmetry about the mean and the approximation 
to a normal distribution. It frequently happens that the new units provide more 
natural expressions of the characteristics being studied. 

Appropriate transformations are suggested by (1) theoretical considerations or 
(2) the data themselves (or both). It has been shown theoretically that data that are 
counts can often be made more normal by taking their square roots . Similarly, the 
logit transformation applied to proportions and Fisher's z-transformation applied to 
correlation coefficients yield quantities that are approximately normally distributed. 

Helpful Transformations To Near Normality 

Original Scale Transformed Scale 

1. Counts, y V? 

2. Proportions ,p logit(p) = ^log ^ f j j (4-33) 

3. Correlations, r Fisher’s z(r) = ~ log 
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In many instances, the choice of a transformation to improve the approximation 
to normality is not obvious. For such cases, it is convenient to let the data suggest a 
transformation. A useful family of transformations for this purpose is the family of 
power transformations. 

Power transformations are defined only for positive variables. However, this is 
not as restrictive as it seems, because a single constant can be added to each obser¬ 
vation in the data set if some of the values are negative. 

Let x represent an arbitrary observation. The power family of transformations 
is in exed by a parameter A. A given value for A implies a particular transformation. 
For example, consider x with A = -1. Since x~ l = 1/jt, this choice of A corre¬ 
sponds to the reciprocal transformation. We can trace the family of transformations 
as A ranges from negative to positive powers of x . For A = 0, we define x° = In x. A 
sequence of possible transformations is 

X_1 = j,*° =lnx,x'/4 = tfi.jtV* = V5, ... 

shrinks large values of x increases large 

values of x 

To select a power transformation, an investigator looks at the marginal dot dia¬ 
gram or histogram and decides whether large values have to be “pulled in” or 
“pushed out” to improve the symmetry about the mean.Trial-and-error calculations 
with a few of the foregoing transformations should produce an improvement. The 
final cho^e should always be examined by a Q~Q plot or other checks to see 
whether the tentative normal assumption is satisfactory. 

The transformations we have been discussing are data based in the sense that it 
is only the appearance of the data themselves that influences the choice of an appro¬ 
priate transformation. There are no external considerations involved, although the 
transformation actually used is often determined by some mix of information sup¬ 
plied by the data and extra-data factors, such as simplicity or ease of interpretation. 

A convenient analytical method is available for choosing a power transforma¬ 
tion. We begin by focusing our attention on the univariate case. 

Box and Cox [3] consider the slightly modified family of power transformations 

fx A -l 

xW A A * ° (4-34) 

(In x A - 0 


which is continuous in A for x > 0. (See [8].) Given the observations x,, * 2 . 

t e Box Cox solution for the choice of an appropriate power A is the solution that 
maximizes the expression 


j; j (,<*>-*»>)’] +(A-1)T to,, 


w e note that xj A) is defined in (4-34) and 



(4-35) 


(4-36) 
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c the arithmetic average of the transformed observations. The first term in (4-35) is, 
S Z frcnna jhe log.ri.bm of a norm.l 

* ^ t with respect to the population mean and vanance parameter . 

' ng iTeca culation of <( A) for many values of A is an easy task for a computer. It* 
The calculation v j j ^ well as a tabular display of the pairs 

u"ea)t°iiTorde.lo study the behavior near the matimiring value A. For ius.auce, 
ifeither A = 0 (logarithm) o, A - \ (square root) knar A. one of these may be pm- 

caleulation of (4-35), some statistician, recommend 
the equivalent procedure of fixing A, creating the new variable 

1 4 


{(s*r" 


7=1,...,* 


and then calculating the sample variance.The minimum of the variance occurs at the 

Sam ^mmr f a ItTnow iunderstood that the transformation obtained by maxima, 

. mTus^allJ proves the approximation to normality. However, there is no 
"Grantee tha eve/the best choice of A will produce a transformed set of values 
fhat adeauate y conform to a normal distnbution. The outcomes produced by a 
fnrmadon selected according to (4-35) should always be carefully examined for 
TsSe vSons of the tentative assumption of normality^* warning appl.es 
with equal force to transformations selected by any other technique. 

I 'u . , 6 (Determining a power transformation for univariate data) We gave 
^Tnos of the microwave radiation emitted through the closed doors of n - 42 
r6ad f Example 4 10 The Q-Q plot of these data in Figure 4.6 indicates that the 
T^vatSs‘Sate from what would be expected if they were normal y d.str.b- 
°td Since all the observations are positive, let us perform a power transformation 
u f d , S " which we hope will produce results that are more nearly normal 
Restricting our'attention il the family of tt.nsfomtations in (4-34), w. mus, Cud 
that value of A maximizing the function €(A) in (4-35). 

The pairs (A, £( A)) are list ed in the following table for several v alues of A. 


70.52 

75.65 

80.46 

84.94 

89.06 

92.79 

96.10 

98.97 

101.39 

103.35 

104.83 

105.84 


106.20 

105.50 

104.43 

103.03 

101.33 

99.34 

97.10 
94.64 
91.96 

89.10 
86.07 
82.88 
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€(A) 



0.0 0.1 0.2 j 0.3 0.4 0.5 


A = 0.28 

Figure 4.12 Plot of £(A) versus A for radiation data (door closed). 


The curve of €(\) versus A that allows the more exact determination A = .28 is 
shown in Figure 4.12 . 


It is evident from both the table and the plot that a value of A around .30 
maximizes €(A). For convenience, we choose A = .25. The data xj were 
reexpressed as 


X V 4 - 1 

r (i/4) = ti _i 

j i 


; = 1,2,...,42 


4 


and a Q-Q plot was constructed from the transformed quantities.This plot is shown 
in Figure 4.13 on page 196.The quantile pairs fall very close to a straight line, and we 
would conclude from this evidence that the are approximately normal. ■ 


Transforming Multivariate Observations 

With multivariate observations, a power transformation must be selected for each of 
the variables. Let Aj, A 2 ,..., X p be the power transformations for the p measured 
characteristics. Each X k can be selected by maximizing 
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r(iw) 
■* ij) 



Figure 4 13 A Q-Q plot of the transformed radiation data (door closed), 
friers in the plot indicate the number of points occupying the same 


location.) 



, r v , are the n observations on the kt h variable, k 

where x xk , Xnk 1 

Here 



1, 2,..., p. 

(4-39) 


is the arithmetic average of the transformed observations The )th transformed mul- 
tivariate observation is 



where , A 2 , 


A are the values that individually maximize (4-38). 
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The procedure just described is equivalent to making each marginal distribution 
approximately normal. Although normal marginals are not sufficient to ensure that 
the joint distribution is normal, in practical applications this may be good enough. 
If not, we could start with the values A 2 , A 2 ,..., A p obtained from the preceding 
transformations and iterate toward the set of values A' = [Ai, A 2 ,..., A p ], which col¬ 
lectively maximizes 

^(^i» A 2 ,..., A p ) 

= ~xln|S(A)| + (A, — 1) 2 In x n + (A 2 - 1) 2 In x j2 
L /=i M 

+ ■■■+ (A, ~ 1) 2 Inxjp (4-40) 

y=i 

where S(A) is the sample covariance matrix computed from 



Maximizing (4-40) not only is substantially more difficult than maximizing the indi¬ 
vidual expressions in (4-38), but also is unlikely to yield remarkably better results. The 
selection method based on Equation (4-40) is equivalent to maximizing a multivariate 
likelihood over /*, X and A, whereas the method based on (4-38) corresponds to maxi¬ 
mizing the kth univariate likelihood over jjL k ,cr kk , and A fc . The latter likelihood is 
generated by pretending there is some A* for which the observations (xfy - 1)/A*, 
j = 1, 2,..., n have a normal distribution. See [3] and [2] for detailed discussions of the 
univariate and multivariate cases, respectively (Also, see [8].) 


Example 4.17 (Determining power transformations for bivariate data) Radiation 
measurements were also recorded through the open doors of the n — 42 
microwave ovens introduced in Example 4.10. The amount of radiation emitted 
through the open doors of these ovens is listed in Table 4.5. 

In accordance with the procedure outlined in Example 4.16, a power transfor¬ 
mation for these data was selected by maximizing €(A) in (4-35). The approximate 
maximizing value was A = .30. Figure 4.14 on page 199 shows Q-Q plots of the un¬ 
transformed and transformed door-open radiation data. (These data were actually 
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transformed by taking the fourth root, as in Example 4.16.) It is clear from the figure 
that the transformed data are more nearly normal, although the normal approxima¬ 
tion is not as good as it was for the door-closed data. 

Let us denote the door-closed data by xu, x 21 ,..., x 42t i and the door-open data 
by * 12 * * 22 » • • •, * 42 , 2 - Choosing a power transformation for each set by maximizing 
the expression in (4-35) is equivalent to maximizing 4(A) (4*38) with k = 1,2. 

Thus, using the outcomes from Example 4.16 and the foregoing results, we have 
A x = .30 and A 2 = .30. These powers were determined for the marginal distribu¬ 
tions of Xi and x 2 . 

We can consider the joint distribution of x x and x 2 and simultaneously deter¬ 
mine the pair of powers (A x , A 2 ) that makes this joint distribution approximately 
bivariate normal. To do this, we must maximize €(Ai, A 2 ) in (4-40) with respect to 
both A x and A 2 . 

We computed €(A], A 2 ) for a grid of A 1t A 2 values covering 0 < A x s .50 and 
0 < A 2 ^ .50, and we constructed the contour plot shown in Figure 4.15 on 
page 200. We see that the maximum occurs at about (A x , A 2 ) = (.16, .16). 

The “best” power transformations for this bivariate case do not differ substan¬ 
tially from those obtained by considering each marginal distribution. ■ 

As we saw in Example 4.17, making each marginal distribution approximately 
normal is roughly equivalent to addressing the bivariate distribution directly and 
making it approximately normal. It is generally easier to select appropriate transfor¬ 
mations for the marginal distributions than for the joint distributions. 
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(b) 

Figure 4*14 Q-Q plots of (a) the original and (b) the transformed 
radiation data (with door open). (The integers in the plot indicate the 
number of points occupying the same location.) 
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Figure 4.15 Contour plot of €(Aj, A 2 ) for the radiation data. 


If the data includes some large negative values and have a single long tail, a 
m ore general transformation (see Yeo and Johnson [14]) should be applied. 

f {(jr + 1) A - 1}/A x a 0, A + 0 

(A) U(* + D x a 0, A = 0 

* j -{(-x + 1) 2 ' A - l}/(2 - A) x < 0,A 4= 2 

t —In (—x + 1) x < 0, A = 2 

Exercises --- 

4.1. Consider a bivariate normal distribution with mi = 1, M2 = 3, tr„ = 2, a 22 = 1 and 
Pl2 = 

(a) Write out the bivariate normal density. ^ 

(b ) Write out the squared statistical distance expression (x - #0'*" (* “ #0 asaqua- 

dratic function of x Y and x 2 . 

4.2. Consider a bivariate normal population with \x j = 0, fx 2 = 2, <j u “ 2 > a 22 - 3X16 

Pi 2 = 

(a) Write out the bivariate normal density. 
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(b) Write out the squared generalized distance expression (x - /*)'X *(* - /i) as a 
function of Xj and x 2 . 

(c) Determine (and sketch) the constant-density contour that contains 50% of the 
probability. 

4.3. Let X be N 3 (fi y X) with ft' = [—3, 1, 4] and 

1 -2 0 " 

- X = -2 5 0 

0 0 2 _ 

Which of the following random variables are independent? Explain. 

(a) Xj and X 2 

(b) X 2 and X 3 

(c) (X U X 2 ) and X 3 
X 1 4 - X 2 

(d) — l 2 —and Y 3 

(e) X 2 and X 2 ~\x l - X 2 

4.4* Let X be 7V 3 (/t, X) with ft' - [2,-3,1] and 

"l 1 r 

X - 1 3 2 

,1 2 2 _ 

(a) Find the distribution of 3A"i - 2A" 2 + X 3 . 

(b) Relabel the variables if necessary, and find a 2 X 1 vector a such that X 2 and 

X 

X 2 - a' are independent. 

4.5. Specify each of the following. 

(a) The conditional distribution of X u given that X 2 — x 2 for the joint distribution in 
Exercise 4.2. 

(b) The conditional distribution of X 2 , given that X\ = xi and X 3 — x 3 for the joint dis¬ 
tribution in Exercise 4.3. 

(c) The conditional distribution of X 3 , given that Xi = x Y and X 2 = x 2 for the joint dis¬ 
tribution in Exercise 4.4. 

4 . 6 . Let X be distributed as , X), where ' = [1,-1,2] and 

4 0 -l" 

X = 0 5 0 

_-l 0 2^ 

Which of the following random variables are independent? Explain. 

(a) Xi and * 2 

(b) Xi and X 3 ’ 

(c) X 2 and * 3 

(d) (Xi t X 3 )andX 2 

(e) Xi and X x + 3X 2 - 2* 3 
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4.7. Refer to Exercise 4.6 and specify each of the following. 

(a) The conditional distribution of X,, given that X 3 = *3- 

(b) The conditional distribution of X,, given that X 2 = *2 and x 3 ~ *3- 

8 (Example of a nonnormal bivariate distribution with normal marginals.) Letjy.be- 
N(0, 1 ), and let' 

j-X, if-lsX.sl 
Al \ otherwise 

Show each of the following. 

(a) X 2 also has an N(0,1) distribution. 

(b) X, and X 2 do not have a bivariate normal distribution. 

„ ■ xun ii p[-1 < Y, < r] = P[-x == X, < 1] for any x. When 

<•> s »“ >•/'] Vpf*,V-n. n-i < i. * i.1 - w * -1] 

= + P[-*is V, < l].Bme[-jc 2 £ V, < 1J 

+ p -1 < X. s x 2 ] from the symmetry argument in the first line of this hint. 

Thus! p\%2 ^' x 2 ] -"I! + ^[-1 < " JC2] ' Whlch,S 

a standard normal probability. ..... 

(b ) Consider the linear combination X, - X 2 . which equals zero with probabthty 
P[|Xi| > 1] = - 3174 - 

4 9 Refer to Exercise 4.8, but modify the construction by replacing the break point 1 by 
c so that 

_ if —c ^ X x ^ c 

Xl ~ \ X\ elsewhere 

Show that c can be chosen so that Cov(X„ X 2 ) = 0. but that the two random variables 
are not independent. 

Ftofc = 0, evaluate Cov(X,, X 2 ) , 

For c very large,evaluate Cov (A^, X 2 ) - E[Xd X x )\. 

4 .IO. Show each of the following. 

( a ) I a n I 


= IAIIB | for |A|*0 


A 0 A Ojjl 0j Expandin g t he determinant R | by the first row 

*’ n„f.nition 2A24) gives 1 times a determinant of the same form, with the order 
oTlreduced by onaThisprocedure is repeated until 1 X |B| is obtained. Sumlarly. 

expanding the determinant | J ®| by the last row gives |* \ =|A|. 
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A 

C 

A 

0 

I A~'C 

0' 

B 

O' 

B 

0' I 


. But expanding the determinant 


I A‘C 
0’ I 


I A _1 C 

by the last row gives ^ 


= 1. Now use the result in Part a. 


4.11. Show that, if A is square, 

|A| = |A 22 ||A U — A 12 A 22 A 21 I for|A22| ^ 0 
= I A n ||A 22 ~ A^A^A^| for | A n | ^ 0 
Hint: Partition A and verify that 


-A 12 A 22 An A 12 

i J |_A 2 j a 22 


I 0 A u - A! 2 A 22 A 2 1 
_-ajJa 21 IJ L •' 


Take determinants on both sides of this equality. Use Exercise 4.10 for the first and 
third determinants on the left and for the determinant on the right. The second equality 
for | A | follows by considering 


I oITa,, a 12 1[i -a^ 

■ 21 A 11 I_ _A 21 A 22 _ O’ I 


1 12 _ All v 

J \ A 22 “ A 21 Ai}A l2 


4.12. Show that, for A symmetric. 


A -i r 1 oir(A; 1 -A 12 A^A 2 ir 1 o n r i 

_ A 22 A 2 1 Ij [_ 0’ 

Thus, (A n - Aj 2 A 2 2 A 2 ])“ 1 is the upper left-hand block of A" 1 . 


-a 12 a 22 

I 


Hint: Premultiply the expression in the hint to Exercise 4.11 by 


postmultiply by 


i 0T 1 

_-aj 2 a 21 I 


I "A 12 A 22 

O' I J 


Take inverses of the resulting expression. 


4.13. Show the following if X is N p (fJL, X) with | X | * 0. 

(a) Check that \X\ = |£ 22 ||£ii ~ ^ 12 ^ 22^21 1- (Note that \X\ can be factored into 
the product of contributions from the marginal and conditional distributions.) 

(b) Check that 

(x - m)'E _1 (* - m) = [*1 - Ml - 2 12 Z 2 -!(x 2 - M 2 )]' 

* (Eli ~ Ei 2 £ 22 Z 2 i) *[x, - Mi — Ei 2 X 22 (x 2 — M 2 )] 
+ (*2 ~ M 2 )'EJ 2 (*2 - M 2 ) 


(Thus, the joint density exponent can be written as the sum of two terms corresponding 
to contributions from the conditional and marginal distributions.) 

(c) Given the results in Parts a and b, identify the marginal distribution of X 2 and the 
conditional distribution of X 2 1 X 2 = x 2 . 
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Hint: 

(a) Apply Exercise 4.11. , 

(b) Note from Exercise 4.12 that we can write (x - m)' 2 (* “ M) as 



If we group the product so that 


II we group me pwuvi w > . 

(”l -1,2^22 X 1 - Ml _ *1 “ Ml “ ^12^22(X2 “ M 2 ) 

0' I J |_ x 2 " M2_] L *2 - M2 - 

the result follows. 

4 .1 4 . If X is distributed as N„(m ,2) with | X | * 0, show that the joint density can be written 
as the product of marginal densities for 


Xi and X 2 if ^12 - 0 

(,xi) ((p-fl)xl) (^x(p-fl)) 


Hint Show by block multiplication that 


r*i 

|_ 0 ' X 22 _ 


the inverse of X 


Then write 


0 

O’ X 22 


.,Dii 0 


„ , V1 ill » *1 - Mi 

(x - m)'2 -1 (* - M) - [(*i - Mi) -(*2 - Ml) l|_ 0 - S -iJ [ X2 - M2 _ 

= (X( - Ml)'^Il( X l " Ml) + ( x 2 - M2)'^2K X 2 - M2) 

Note that | X | - | Xi 1 1| Xn I from Exercise 410(a) - NoW faCtOT the j ° int density ' 

4-.5. Show that t (x/ - x)(x - m)' and | (* - M)( x ; - x)’ are both p X p matrices of 
/ = 1 * 

zeros. Here x'- = [*;i»*j2» • ■ ■ * J ” 1,2,..., n, and 

1 " 

x = - 2 x i 
* / -1 

4 16 Let X [ , X 2 , X 3 , and X 4 be independent N p (m, 2) random vectors. 

(a) Find the marginal distributions for each of the random vectors 

v, = lx, - jX 2 + ;X 3 - Jx 4 


2 = lx, + ^x 2 - Jx 3 - }x 4 


(b) Find the joint density of the random vectors V! and V 2 defined in (a). 

117 LetX, X, X,X 4 and X 5 be independent and identically distributed[random vectors 
4 with^mean 2 vectorpand covariance matrix X. Find die mean vector and covariance ma¬ 

trices for each of the two linear combinations of random vectors 


ix, + ^x 2 + |X 3 + gX 4 + 5X5 
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and 

Xj - X 2 + X 3 - X 4 + X 5 

in terms of /z and X. Also, obtain the covariance between the two linear combinations of 
random vectors. 

4-18. Find the maximum likelihood estimates of the 2X1 mean vector /z and the 2X2 
covariance matrix X based on the random sample 



from a bivariate normal population. 

4.19. Let Xi,X 2 ,..., X 20 be a random sample of size n = 20 from an X) population. 

Specify each of the following completely. 

(a) The distribution of (X! - /a)'X _1 (X 1 - ft) 

(b) The distributions of X and Vn(X - ft) 

(c) The distribution of (n — 1) S 

4.20. For the random variables X lt X 2 ,... ,X 2U in Exercise 4.19, specify the distribution of 
B(19S)B' in each case. 



0 0 0~ 
_! i 

2 2 1 J 


(b) B = 


1 0 0 0 0 (f 

0 0 1 0 0 0 _ 


4.2 1 . Let Xi,..., X 6 o be a random sample of size 60 from a four-variate normal distribution 
having mean /z and covariance X. Specify each of the following completely. 

(a) The distribution of X 

(b) The distribution of (X! — /i)'X" 1 (X 1 - ft) 

(c) The distribution of n(X — pi)'X" 1 (X - /z) 

(d) The approximate distribution of n(X - /z}'S _1 (X - ft) 

4.22. Let Xj, X 2 ,..., X 75 be a random sample from a population distribution with mean /z 
and covariance matrix X. What is the approximate distribution of each of the following? 

. (a) X _ 

(b) n(X - /tyS'HX-p) 

4.23. Consider the annual rates of return (including dividends) on the Dow-Jones 
industrial average for the years 1996-2005. These data, multiplied by 100, are 

-0.6 3.1 25.3 -16.8 -7.1 -6.2 25.2 22.6 26.0. 

Use these 10 observations to complete the following. 

(a) Construct a Q-Q plot. Do the data seem to be normally distributed? Explain. 

(b) Carry out a test of normality based on the correlation coefficient r Q . [See (4-31).] 
Let the significance level be a = .10. 

4.24. Exercise 1.4 contains data on three variables for the world’s 10 largest companies as of 
April 2005. For the sales (j^) and profits (x 2 ) data: 

(a) Construct Q-Q plots. Do these data appear to be normally distributed? Explain. 
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Carrv out a test of normality based on the correlation coefficient r Q [See (4-31).] 

(b) get the significance level at a = .10. Do the results of these tests corroborate the re- j 

suits in Part a? 

4 25 Refer to the data for the world’s 10 largest companies in Exercise 1.4. Construct a chi- 
Square plot using all three variables.^ ch,-square quantiles are 

0.3518 0.7978 1.2125 1.6416 2.1095 2.6430 3.2831 4.1083 5.3170 7.8147 
, ^ . i o „;vpc the ace jo measured in years, as well as the selling price x 2 , measured 

4 -“- - io— -w «-t”— - 


*1 

1 

2 

3 34 3 6 5V ii 

*2 

18.95 

19.00 

' 17.95 15.54 14.00 12.95 8.94 7.49 6.00 3.99 


Use the results of Exercise 1.2 to calculate the squared statistical distances 

_ x)'S _1 (x- — x),/ = 1,2.10, where [jc/i>-^ 2 ]- 

, L , ,,'. ,; c ,' re , in Part a, determine the proportion of the observations falling 

(b) Sf n the estimated 50% probability contour of a bivariate normal distribution. 

(c) Order the distances in Part a and construct a chi-square plot 

(d) Given the results in Parts b and c, are these data approximately b.vanate normal? 
Explain. 

_ • a ^Hiotion data (with door closed) in Example 4.10. Construct a Q-Q plot 

4.27. Consider the radi f ,hesedata [Note that the natural logarithm transformation 

“ilfi' "tS (S>1 «» logarithms appear ,o be jot- 

Compare »«u, Ults with Ftgare 4.13. Doe, the choice A - » « 

A = 0 make much difference in this case. 

The following exercises may require a computer. 

.. „ir dilution data given in Table 1.5. Construct a Q-Q plot for the solar 

4.28. Consider the ‘P d carry ou , a test for normality based on the correlation 

coefSnt m re [see (4-31)]. Let a = .05 and use the entry corresponding to n - 40 in 

Table 4.2. _ 

4 29 Given the air-pollution data in Table 1.5, examine the pairs X 5 = N0 2 and - Oa for 

W distances C, - - «>. - = ..2.«. "her. 

(k> St!bi“wS» probah!“ly conltor lla'bltaiia'te normal dUtribntion. 
(c) Construct a chi-square plot of the ordered distances in Part a. 

4 30. Consider the used-car data in Exercise 4.26 ; 

(a) Determine the power transformation A, that makes the x, values approximately 
normal. Construct a Q-Q plot for the transformed data, 
fb) Determine the power transformations A 2 that makes the x 2 values approxunately 
( > normaTconstruct a Q-Q plot for the transformed data. 

, . Dptermine the power transformations A' = [A„A 2 ] that make the [x,, x 2 ] values 
C jointly normal iing (4-40). Compare the results with those obtained in Parts a and b. 
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4.31. Examine the marginal normality of the observations on variables X x , X 2 ,..., X s for the 
multiple-sclerosis data in Table 1.6. Treat the non-multiple-sclerosis and multiple-sclerosis 
groups separately. Use whatever methodology, including transformations, you feel is 
appropriate. 

4-32. Examine the marginal normality of the observations on variables X x , X 2y ..., X 6 for the 
radiotherapy data in Table 1.7. Use whatever methodology, including transformations, 
you feel is appropriate. 

4.33. Examine the marginal and bivariate normality of the observations on variables 
X x , X 2 y X 3 , and X 4 for the data in Table 4.3. 

4.34. Examine the data on bone mineral content in Table 1.8 for marginal and bivariate nor¬ 
mality. 

4.35. Examine the data on paper-quality measurements in Table 1.2 for marginal and multi¬ 
variate normality. 

4-36. Examine the data on women’s national track records in Table 1.9 for marginal and mul¬ 
tivariate normality. 

4.37. Refer to Exercise 1.18. Convert the women’s track records in Table 1.9 to speeds mea¬ 
sured in meters per second. Examine the data on speeds for marginal and multivariate 
normality. 

4.38. Examine the data on bulls in Table 1.10 for marginal and multivariate normality. Consider 
only the variables YrHgt, FtFrBody, PrctFFB, BkFat, SaleHt, and SaleWL 

4.39. The data in Table 4.6 (see the psychological profile data: www.prenhall.com/statistics) con¬ 
sist of 130 observations generated by scores on a psychological test administered to Peru¬ 
vian teenagers (ages 15,16, and 17). For each of these teenagers the gender (male = 1, 
female = 2) and socioeconomic status (low = 1, medium = 2) were also recorded. The 
scores were accumulated into five subscale scores labeled independence (indep), support 
(supp), benevolence (benev), conformity (conform), and leadership (leader). 


Table 4.6 Psychological Profile Data 

Indep 

Supp 

Benev 

Conform 

Leader 

Gender 

Socio 

27 

13 

14 

20 

11 

2 

1 

12 

13 

24 

25 

6 

2 

1 

14 

20 

15 

16 

7 

2 

1 

18 

20 

17 

12 

6 

2 

1 

9 

22 

22 

21 

6 

2 

1 

10 

11 

26 

17 

10 

1 

2 

14 

12 

14 

11 

29 

1 

2 

19 

11 

23 

18 

13 

2 

2 

27 

19 

22 

7 

9 

2 

2 

10 

17 

22 

22 

8 

2 

2 

Source: Data courtesy of C. Soto. 


(a) Examine each of the variables independence, support, benevolence, conformity and 
leadership for marginal normality. 

(b) Using all five variables, check for multivariate normality. 

(c) Refer to part (a). For those variables that are nonnormal, determine the transformation 
that makes them more nearly normal. 
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4.40. Consider the data on national parks in Exercise 1.27. 

(a) Comment on any possible outliers in a scatter plot of the original variables. 

(b) Determine the power transformation A] the makes the X\ values approximately 
normal. Construct a Q-Q plot of the transformed observations. 

(c) Determine The power transformation A 2 the makes the x 2 values approximately 
normal. Construct a Q-Q plot of the transformed observations. 

(d) Determine the power transformation for approximate bivariate normality usings 

(4-40). 1 

4 . 41 . Consider the data on snow removal in Exercise 3.20. -J 

(a) Comment on any possible outliers in a scatter plot of the original variables. J| 

(b) Determine the power transformation Aj the makes the x 1 values approximately ! 

normal. Construct a Q-Q plot of the transformed observations. * 

(c) Determine the power transformation A 2 the makes the x 2 values approximately 
normal. Construct a Q-Q plot of the transformed observations. 

(d) Determine the power transformation for approximate bivariate normality usings 
(4-40). 
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Chapter 



Inferences about a Mean Vector 

5.1 Introduction 


m 


This chapter is the first of the methodological sections of the book. We shall now use 
the concepts and results set forth in Chapters 1 through 4 to develop techniques for 
analyzing data. A large part of any analysis is concerned with inference —that is, 
reaching valid conclusions concerning a population on the basis of information from a 
sample. 

At this point, we shall concentrate on inferences about a population mean 
vector and its component parts. Although we introduce statistical inference through 
initial discussions of tests of hypotheses, our ultimate aim is to present a full statisti¬ 
cal analysis of the component means based on simultaneous confidence statements. 

One of the central messages of multivariate analysis is that p correlated 
variables must be analyzed jointly. This principle is exemplified by the methods 
presented in this chapter. 



5.2 The Plausibility of p 0 as a Value for a Normal 
Population Mean 


Let us start by recalling the univariate theory for determining whether a specific value 
Pq is a plausible value for the population mean p. From the point of view of hypothe¬ 
sis testing, this problem can be formulated as a test of the competing hypotheses 


H 0 : p = p Q and H } : p * p Q 


Here H 0 is the null hypothesis and Hi is the (two-sided) alternative hypothesis. If 
X x , X 2 ,.. -, X n denote a random sample from a normal population, the appropriate 
test statistic is 


t 


(T-Mo) 

s/Vn 


_ \ n 

where X — — 2 Xj and 

n jm i 


n 1 y =1 


X) 2 


4 


i 
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This test statistic has a student’s ^-distribution with n — 1 degrees of freedom (d.f.). 
We reject Hq , that pq is a plausible value of p y if the observed 1 1 1 exceeds a specified 
percentage point of a r-distribution with n - 1 d.f. 

Rejecting Hq when 1 1 | is large is equivalent to rejecting Hq if its square, 

, (X - Pq) 2 _ , _ 

' * s 2 /n = «(* - ^)(S 2 )~\X - p Q ) (5-1) 

is large. The variable t 2 in (5-1) is the square of the distance from the sample mean 
X to the test value pq, The units^of distance are expressed in terms of s/Vn> or esti¬ 
mated standard deviations ofX. Once X and s 2 are observed, the test becomes: 
Reject H q in favor of H l , at significance level a, if 

”(* ~ Mo )(s 2 )'\x - pq) > t 2 _i{a/ 2) (5-2) 

where t n -\(<x/2) denotes the upper 100(a/2)th percentile of the r-distribution with 
n - 1 d.f. 

If Hq is not rejected, we conclude that pq is a plausible value for the normal 
population mean. Are there other values of p which are also consistent with the 
data? The answer is yes! In fact, there is always a set of plausible values for a nor¬ 
mal population mean. From the well-known correspondence between acceptance 
regions for tests of H Q : p = p^ versus H\. p ^ pq and confidence intervals for p, 
we have 


{Do not reject H 0 : p - p^ at level a} or 
is equivalent to 


x ~ Mo 


s/Vn 


'n-1(«/2) 


|/io lies in the 100(1 - a) % confidence interval x ± r„_j(a/2)^= j 


or 


* - t n ^(a/2) ~^= < pq < x 4- t n _i{a/2)^= (5-3) 

The confidence interval consists of all those values pq that would not be rejected by 
the level a test of H 0 : p - pq. 

Before the sample is selected, the 100(1 - a)% confidence interval in (5-3) is a 
random interval because the endpoints depend upon the random variables^ and s. 
The probability that the interval contains p is 1 — a; among large numbers of such 
independent intervals, approximately 100(1 - a)% of them will contain p. 

Consider now the problem of determining whether a given p X 1 vector /jlq is a 
plausible value for the mean of a multivariate normal distribution. We shall proceed 
by analogy to the univariate development just presented. 

A natural generalization of the squared distance in (5-1) is its multivariate analog 

T 2 = (X - Mo)'(is) (X - Mo ) = n(X - Mo)'S"'(X - Mo) (5-4) 
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1 n 

X =-2x y , 

(pxi) n > 


S = 

(pxp) 


r2(X-X)(X y -X)',and mo 

« 1 y—! (pXl) 


The statistic T 2 is called Hotelling's T 2 in honor of Harold Hotelling, a pioneer in 
multivariate analysis, who first obtained its sampling distribution. Here (l/n)S is the 
estimated covariance matrix of X. (See Result 3.1.) 

If the observed statistical distance T 2 is too large—that is, if x is “too far” from 
Mo—the hypothesis H 0 : /x = is rejected. It turns out that special tables of T 2 per¬ 
centage points are not required for formal tests of hypotheses. This is true because 

9 (n - 1 )p 

T 2 is distributed as—-(5-5) 

("-/>) 

where F p n _ p denotes a random variable with an F-distribution with p and n - p d.f 
To summarize, we have the following: 


Let X 1} X 2> ..., X„ be a random sample from an N p (/x , X) population. Then 
wth X = i 2 x > and S = y-T— 2 (Xy - X) (X y - X)', 

n j =l ” -U y=] 

~ = p r^2 i? 


“ P [_ T > (n- p) 

= P n(X - - fi) > ^ (5-6) 

whatever the true /a and X. Here F p , n ~ p (ct) is the upper (100a)th percentile of 
the F p n _ p distribution. 


Statement (5-6) leads immediately to a test of the hypothesis H {) : p = /x 0 versus 
//i: pi t* /bt 0 . At the a level of significance, we reject Pf 0 in favor of H] if the 
observed 

T 2 = n(x - Mo)'S _1 (x - Mo) > ^ F p _„- p (a) (5-7) 

It is informative to discuss the nature of the redistribution briefly and its cor¬ 
respondence with the univariate test statistic. In Section 4.4, we described the man¬ 
ner in which the Wishart distribution generalizes the chi-square distribution. We 
can write 


f 2 (x y - x)(x y - X)' | 

T 2 =Vh(X- mo)' —-T- VS(X- Mo) 
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which combines a normal, N p ( 0, X), random vector and a Wishart, W„ __,(X), random 
matrix in the form 


T 2 - 

1 p, n — J - 


multivariate normal 
random vector 


Wishart random 
matrix 
df. 


multivariate normal 
random vector 


wz) '[^ r r w ' , - n - i(2) J w 2 ) 


This is analogous to 


t 2 = VS (X - Mo)( j2 ) - 'V« (X - Mo) 


normal 

random variable 


(scaled) chi-square 
random variable 
d.f. 


normal 

random variable 


for the univariate case. Since the multivariate normal and Wishart random variables 
are independently distributed [see (4-23)], their joint density function is the product 
“ marginal normal and Wishart distributions. Using calculus, the distribution 
p-b) of T as given previously can be derived from this joint distribution and the 
representation (5-8). 

It is rare, in multivariate situations, to be content with a test of H 0 : u = u 0 , 
where all of the mean vector components are specified under the null hypothesis’ 
Ordinarily, it is preferable to find regions of m values that are plausible in light of 
the observed data. We shall return to this issue in Section 5.4. 


Example s. I (Evaluating T 2 ) Let the data matrix for a random sample of size n = 3 
from a bivariate normal population be 

”6 9~ 

X = 10 6 

- 8 3_ 

Evaluate the observed T 2 for Mo = [9, 5], What is the sampling distribution of T 2 in 
this case? We find 

6 + 10 + 8 ' 

3 

9 + 6 + 3 
3 

and 

_ _ (6 - 8) 2 + (10 - 8) 2 + (8 - 8) 2 
-j-“ = 4 

= (6 ~ 8)(9 - 6) + (10 - 8)(6 - 6) + (8 - 8)(3 - 6) 

12 ~ 2 

_ _ (9 - 6) 2 + (6 - 6) 2 + (3 -r 61 2 
*22- - - — = 9 
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(4)(9) - ( 3)( 3) |_3 


and, from (5-4), 


I £JL 6-5 


— 3 [— 1 , 1 ] 


Before the sample is selected, T 2 has the distribution of a 

£? Z . 1 ) 2 p 2 = 4 F 2 i 
(3 - 2) 2,3 

random variable. 

The next example illustrates a test of the hypothesis H 0 : M = Mo using data 
collected as part of a search for new diagnostic techniques at the University of 
Wisconsin Medical School. 

Example 5.2 (Testing a multivariate mean vector with T 2 ) Perspiration^ from 20 
healthy females was analyzed. Three components, X 3 = sweat rate, - sodium 
content, and X 3 = potassium content, were measured, and the results, which we call 

the sweat data, are presented in Table 5.1. 

Test the hypothesis H,y. ft' = [4, 50,10] against Hy. m' * [4, 50,10] at level of 

significance a = .10. 


Computer calculations provide 

f 4.6401 


10.010 

199.788 

-5.640 


-1.810 

-5.640 

3.628 


We evaluate 


20 [4.640 - 4, 45.400 - 50, 9.965 - 10] 


.586 -.022 .258 1 4.640 - 4 

-.022 .006 -.002 45.400 - 50 

.258 -.002 .402 L 9 - 965 " 10 


= 20 [ .640, -4.600, -.035] -.042 = 9.74 

.160 I 
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Comparing the observed T 2 = 9.74 with the critical value 
(n - 1 )p 19(3) 

(n - p) F ^-P ( - 10 > = -YT F 3,n(-10) = 3.353(2.44) = 8.18 

we see that T 2 = 9.74 > 8.18, and consequently, we reject H 0 at the 10% level of 
significance. 

We note that H 0 will be rejected if one or more of the component means, or 
some combination of means, differs too much from the hypothesized values 
[4, 50,10]. At this point, we have no idea which of these hypothesized values may 
not be supported by the data. 

We have assumed that the sweat data are multivariate normal. The Q-Q plots 
constructed from the marginal distributions of X lf X 2 , and X 3 all approximate 
straight lines. Moreover, scatter plots for pairs of observations have approximate 
elliptical shapes, and we conclude that the normality assumption was reasonable in 
this case. (See Exercise 5.4.) ■ 

One feature of the T 2 -statistic is that it is invariant (unchanged) under changes 
in the units of measurements for X of the form 

Y = C X + d , 

<pxi) (pxp)(/>xi) (pxi) 


C nonsingular 


(5-9) 
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A transformation of the observations of this kind arises when a constant b i i s 
subtracted from the ith variable to form X x - fr, and the result is multiplied 
by a constant a t > 0 to get a^X-, - b t ). Premultiplication of the centered and 
scaled quantities a / (A r ,- - b,) by any nonsingular matrix will yield Equation ( 5 - 9 ) 
As an example, the operations involved in changing X t to a^Xi - bi) correspond^ 
exactly to the process of converting temperature from a Fahrenheit to a Celsius 
reading. 

Given observations x x , x 2 ,..., x„ and the transformation in (5-9), it immediately 
follows from Result 3.6 that 

y = Cx + d and S y = -±- ± (y y - y) (y, - y)' = CSC' 

n 1 y=l 

Moreover, by (2-24) and (2-45), 

My = E(X) = E( CX + d) = E(CX) + E( d) = Cm + d 
Therefore, T 2 computed with the y’s and a hypothesized value my,o = Cmo + d is 
T 2 = n(y - /iv.oJ'SyHy “ Mv.o) 

= n(C(x - Mo ))'(CSC')“ 1 (C(x - mo)) 

= n(x - Mo)’C'(CSC') *C(x - Mo) 

= n(x - MoJ'C'CC'J^S^C^CCx - Mo) = «(* “ Mo)'S~’( x _ Mo) 

The last expression is recognized as the value of T 2 computed with the x’s. 


5.3 Hotelling’s 7 2 and Likelihood Ratio Tests 

We introduced the T 2 -statistic by analogy with the univariate squared distance t 2 . 
There is a general principle for constructing test procedures called the likelihood 
ratio method , and the T 2 -statistic can be derived as the likelihood ratio test of H 0 : 
M = Mo- The general theory of likelihood ratio tests is beyond the scope of this 
book. (See [3] for a treatment of the topic.) Likelihood ratio tests have several 
optimal properties for reasonably large samples, and they are particularly conve¬ 
nient for hypotheses formulated in terms of multivariate normal parameters. 

We know from (4-18) that the maximum of the multivariate normal likelihood 
as m and X are varied over their possible values is given by 


where 


max L(m,X) 


--- e ~npj2 

(27r) np/2 |Xr /2 


( 5 - 10 ) 


2 = 7 Z ( x y “ 5)(*y - *)' and M = * = \ Z x y 


are the maximum likelihood estimates. Recall that fa and X are those choices for m 
and X that best explain the observed values of the random sample. 
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Under the hypothesis H 0 : m = Mo> the normal likelihood specializes to 


MMo.X) (2 7r )"P/2|2|"/2 CXP 




Mo)'X Hxj ~ Mo) 


The mean Mo ls now fixed, but X can be varied to find the value that is “most likely” 
to have led, with mo fixed, to the observed sample. This value is obtained by maxi¬ 
mizing L(mo ? X) with respect to X. 

Following the steps in (4-13), the exponent in L(moi X) may be written as 
~2 2 (*/ “ Mo)'X _1 ( x y - Mo) = " Z tr X _1 ( x y ~ Mo )( x y “ Mo)' 

7“1 1 7=1 _ J 


= -Itr X *( Z ( x y ~ Mo)( x y - Mo)’ 
z L \}=1 


Applying Result 4.10 with B = Z ( x / ~ Mo)( x y ~ Mo)' and b = n/2, we have 


max L(mo> X) =--- e~ n P/ 2 

2 (2-77-)"^ I 


2o = - Z ( x y “ Mo) ( x y “ Mo)' 
n /=i 

To determine whether Mo i s a plausible value of /*, the maximum of L(mo>X) is 
compared with the unrestricted maximum of L(m, X). The resulting ratio is called 
the likelihood ratio statistic. 

Using Equations (5-10) and (5-11), we get 


maxL(Mo.X) f \±\\n /2 

Likelihood ratio = A =-———— = (-U-L ) (5-12) 

itimL(m,X) V|X 0 |/ 

The equivalent statistic A 2/n = |X|/|X 0 | is called Wilks’ lambda. If the 
observed value of this likelihood ratio is too small, the hypothesis H 0 : m = Mo 1S 
unlikely to be true and is, therefore, rejected. Specifically, the likelihood ratio test of 
H$. m = Mo against H x \ ft A Mo rejects H 0 if 


i x i y/ 2 


S ( x y - x )( x y - x )' 

7 = 1 _ 

n 

Z ( x y - Mo)( x y - Mo)' 
y-i 


where c a is the lower (100a)th percentile of the distribution of A. (Note that the 
likelihood ratio test statistic is a power of the ratio of generalized variances.) Fortu¬ 
nately, because of the following relation between T 2 and A, we do not need the 
distribution of the latter to carry out the test. 
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Result 5.1. Let X,, X 2 ,... , X„ be a random sample from an N p (p, 2) population. 
Then the test in (5-7) based on T 2 is equivalent to the likelihood ratio test of 
H 0 : m = Mo versus m *■ Mo because 


a2/ " H 1 + (^T) 


Proof. Let the (p + 1) x (p + 1) matrix 


E (*/ - *)(*/ ~ *)' 

y=i. 

Vn (x - Mo)' 


Vn (x - Mo) _ A 

—.-J ‘ La 


A u 

An' 

,A 2 i 

A22, 


By Exercise 4.11, | A | — | A 22 11 i A 1 2 A 22 A 2 i| |Au||A22 A 21 A 11 A 12 I, 

from which we obtain 


(-1) 2 (Xy - x)(Xy - x)' + n(x - Mo) ( x “ Mo)' 
7 = 1 


= 2 (*>-*) (*>-*)' -1 -«(* -Mo)' (2 (*;-*)(*/- *)') (*-Mo) 

/=! V l =1 


I 7-1. 

Since, by (4-14), 


V (xy - Mo) (*/ - Mo)' = S (X; - 5 + X - Mo)(*t - 5 + X - mo)' 
i-t ' =1 

= y (xy - x) (Xy - x)' + n(x - Mo) (x - Mo)' 
the foregoing equality involving determinants can be written 

n ( y2 

(-1) i (*/ - Mo )(*7 - Mo)' = i (*7 - *)(*7 - *)' ("D + (7^1) 


|nS 0 | = \n2\\l 


(n - 1) 


A 2/ n = J±L- 1 + 


(n-1) 


Here H 0 is rejected for small values of A 2/ " or, equivalently, large values of T 2 . The 
critical values of T 2 are determined by (5-6). 1 
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Incidentally, relation (5-14) shows that T 2 may be calculated from two determi¬ 
nants, thus avoiding the computation of S“\ Solving (5-14) for T 2 , we have 




(n-1) 2 ( x 7 - Mo) ( x y - mo)' 
_ _/=i _ 

y (xy - x) (Xy - x)' 

7=1 




Likelihood ratio tests are common in multivariate analysis. Their optimal 
large sample properties hold in very general contexts, as we shall indicate shortly. 
They are well suited for the testing situations considered in this book. Likelihood 
ratio methods yield test statistics that reduce to the familiar F- and ^-statistics in uni¬ 
variate situations. 


General Likelihood Ratio Method 


We shall now consider the general likelihood ratio method. Let 0 be a vector consist¬ 
ing of all the unknown population parameters, and let L{0) be the likelihood function 
obtained by evaluating the joint density of X 1? X 2 ,..., X rt at their observed values 
x l5 x 2 ,..., x n . The parameter vector 0 takes its value in the parameter set 0. For 
example, in the p-dimensional multivariate normal case. 0' = [/ij,..., fi p , 
^li,..., <T\ p , <7 2 2» ■»• • •» cr^-i /7 , (Tpp\ and 0 consists of the p-dimensional 
space, where - oo < p x < oo ,..., - oo < p p < oo combined with the 
[p{P + l)/2]-dimensional space of variances and covariances such that X is positive 
definite.Therefore, 0 has dimension v — p + p(p + l)/2. Under the null hypothesis 
H 0 : 0 = 0 O , 0 is restricted to lie in a subset 0 O of 0. For the multivariate normal 
situation with /i = /i 0 and X unspecified, 0 O = {pl x — P\^P 2 — Pvo, • ••» P p = 

0 * 11 ,..., <r lp , cr 22 ,cr 2p ,..., cr pp with X positive definite}, so 0 O has 

dimension v Q = 0 4- p(p + l)/2 = p(p + l)/2. 

A likelihood ratio test of H 0 \ 0 e 0 O rejects H 0 in favor of H x : 0 £ @ 0 if 


max L(0) 
max L(0) 


(5-16) 


where c is a suitably chosen constant. Intuitively, we reject H 0 if the maximum of the 
likelihood obtained by allowing 0 to vary over the set 0 O is much smaller than 
the maximum of the likelihood obtained by varying 0 over all values in 0. When the 
maximum in the numerator of expression (5-16) is much smaller than the maximum 
in the denominator, 0 O does not contain plausible values for 0. 

In each application of the likelihood ratio method, we must obtain the sampling 
distribution of the likelihood-ratio test statistic A. Then c can be selected to produce 
a test with a specified significance level a. However, when the sample size is large 
and certain regularity conditions are satisfied, the sampling distribution of -2 In A 
is well approximated by a chi-square distribution. This attractive feature accounts, in 
part, for the popularity of likelihood ratio procedures. 
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Result 5.2. When the sample size n is large, under the null hypothesis // 0 , 

/ max L( 0 )\ 


-2 In A = -2 In 


max L(0) 

0G0 


is, approximately, a xl-v 9 random variable. Here the degrees of freedom are v - p Q 
= (dimension of 0 ) - (dimension of @ 0 ). m 


Statistical tests are compared on the basis of their power , which is defined as the 
curve or surface whose height is P[test rejects // o [ 0 ], evaluated at each parameter 
vector 0 . Power measures the ability of a test to reject H 0 when it is not true. In the * 
rare situation where 0 = 0 O is completely specified under H 0 and the alternative 
consists of the single specified value 0 = 0 l , the likelihood ratio test has the highest 
power among all tests with the same significance level a = P[test rejects H Q \ 0 = 0 O ]. 
In many single-parameter cases (0 has one component), the likelihood ratio test is 
uniformly most powerful against all alternatives to one side of H 0 : 0 = 6 0 . In other 
cases, this property holds approximately for large samples. 

We shall not give the technical details required for discussing the optimal prop¬ 
erties of likelihood ratio tests in the multivariate situation. The general import of 
these properties, for our purposes, is that they have the highest possible (average) 
power when the sample size is large. 


5.4 Confidence Regions and Simultaneous Comparisons 
of Component Means 

To obtain our primary method for making inferences from a sample, we need to ex¬ 
tend the concept of a univariate confidence interval to a multivariate confidence re¬ 
gion. Let 0 be a vector of unknown population parameters and 0 be the set of all 
possible values of 0. A confidence region is a region of likely 0 values. This region is 
determined by the data, and for the moment, we shall denote it by /?(X), where 
X = [X 1# X 2> ..., X n ]' is the data matrix. 

The region i?(X) is said to be a 100(1 ~ a)% confidence region if, before the 
sample is selected, 

F[i?(X) will cover the true 0 ] = 1 - a (5-17) 

This probability is calculated under the true, but unknown, value of 0. 

The confidence region for the mean fx of a p-dimensional normal population is 
available from (5-6). Before the sample is selected, 

p[*(X - M)'S-‘(X - M) s ( " ~ y F,,„-,(<*)] = 1 - a 

whatever the values of the unknown fx and X. In words, X will be within 
[(n - 1 )pF P '„- p (a)/(n - p)] 1/2 

of fx, with probability 1 — a, provided that distance is defined in terms of nS _1 . 
For a particular sample, x and S can be computed, and the inequality 
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within thn^ ^ fit , ^P F p- n ~p( a )/( n p) will define a region /?(X) 

SSvHr f - P ™ Slble Pieter values. In this case, the region will be an 
ellipsoid centered at x. Tins ellipsoid is the 100(1 - a)% confidence region for p. 

dis2!jnSI° regi ° n f ° r the mean of a /’■dimensional normal 

distribution is the ellipsoid determined by all fx such that 

n( * “ M) ' S " ( * - (5-18) 

n ^ ® ~ („ _ i) 2 ( x i ~ *)(*/ - x)' and x,, x 2 ,..., x„ are 

the sample observations. 

n l„„^? i , de v te , rmil ! e whether any no lies within the confidence region (is a 

!t<x - « Vc-lf- ° r W j neCd t0 com P ute the generalized squared distance 
"1* Mol » (x - p Q ) and compare it with [p(n - 1 )/(n - p)lF (a) If the 

~ 1)/(n - WJUZ lisnotTuThecoifi 

15 71? 1S ,. a i nal ° gOUS t0 testin 8 M = Mo versus Hp p * p 0 [see 

the P Test wo Hh C ° nflde J 1Ce r6gi0n ° f (5 * 18) COnsists of a11 vectors for which 
the T -test would not reject H 0 m favor of H x at significance level a 

tor p s: 4, we cannot graph the joint confidence region for p However we can 

determined "" “T™’ “ d “>*' reMve^ThTX % 

A< “ d ei8 '" v " ,ors * ° ,s as ™ <«>• <»'- 

"<* - -") * w-> 

are determined by going 

VXiC/Vi = Va; Vpjn - 1 )F PiK _ p {a)/n{n - p) 

Spso!S n a 8 re the dg “ VeCt0 ” *'• Beginnin 8 at the “"ter x, the axes of the confidence 

± vx l p ^ n ~ ^~ r 7T 

V n(n — p) F P> n ~P^ e » where Se,- = X^ h / = 1,2,..., p (5-19) 
^?axes tlOS ° f thC A ' S Wi ” he ‘ P identify felatiVe amounts of elongation along pairs 


Examp'e 5.3 (Constructing a confidence ellipse for p) Data for radiation from 
microwave ovens were introduced in Examples 4.10 and 4.17. Let 

*1 = Vmeasured radiation with door closed 
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For the n = 42 pairs of transformed observations, we find that 

.0144 .0117“ 


-[-]■ 

s-i = r 2 ° 3j 

s L" 163 - 


.0117 .0146 


018 -163.391 
391 200.228 


The eigenvalue and eigenvector pairs for S are 

Ai = .026, ei = [.704, .710] 

A 2 = .002, e 2 = [-.710, .704] 

The 95% confidence ellipse for p consists of all values (pi,p 2 ) satisfying 


42[.564 - .603 - p 2 ] 


203.018 

-163.391' 

'.564 - pi 

-163.391 

200.228_ 

.603 — p 2 _ 


2(41) 

40 ^2,4o( >05) 


or, since F 2>40 (.05) = 3.23, 

42(203.018) (.564 - z^) 2 + 42(200.228) (.603 - 

- 84(163.391) (.564 - w )(.603 - /x 2 ) « 6.62 

To see whether p' = [.562, .589] is in the confidence region, we compute 

42(203.018) (.564 - ,562) 2 + 42(200.228) (.603 - .589) 2 

- 84(163.391)(.564 - ,562)(.603 - .589) = 1.30 < 6.62 

We conclude that fx' = [.562, .589] is in the region. Equivalently, a test of H 0 : 




.562 

.589 


would not be rejected in favor of H x : p * 


.562 

.589 


at the a = .05 level 


of significance. 

The joint confidence ellipsoid is plotted in Figure 5.1. The center is at 
x' = [,564, .603], and the half-lengths of the major and minor axes are given by 


and 


VT X 


P(n ~ *) 

n(n - p) 


F p .n-p( a ) = vm 


2(41) 

42(40) 


(3.23) = .064 


p(n - 1) 


F p , n -pM = vm 


2(41) 

42(40) 


(3.23) = .018 


n(n-p) *' 

respectively.The axes lie along e[ = [.704, .710] and e 2 = [-.710, .704] when these 
vectors are plotted with x as the origin. An indication of the elongation of the confi¬ 
dence ellipse is provided by the ratio of the lengths of the major and minor axes. 
This ratio is 


2Va7 


- j p( n ~ 

1 V n(n - p) 


~ 1) 


F Pt n~p( a ) 


2VT 2 


I p( n - 1 ) 

V n(n - p) 


F p , n ~p ( a ) 


Va^ 


.161 

.045 


= 3.6 
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2 



Figure 5.1 A 95% confidence 
ellipse for p based on microwave- 
radiation data. 


The length of the major axis is 3.6 times the length of the minor axis. ■ 


Simultaneous Confidence Statements 

While the confidence region n(x — fx)' S -1 (x - /t) s c 2 , for c a constant, correctly 
assesses the joint knowledge concerning plausible values of fx , any summary of con¬ 
clusions ordinarily includes confidence statements about the individual component 
means. In so doing, we adopt the attitude that all of the separate confidence state¬ 
ments should hold simultaneously with a specified high probability. It is the guaran¬ 
tee of a specified probability against any statement being incorrect that motivates 
the term simultaneous confidence intervals. We begin by considering simultaneous 
confidence statements which are intimately related to the joint confidence region 
based on the T 2 -statistic. 

Let X have an N p (p , X) distribution and form the linear combination 
Z = a x X x T a 2 X 2 + * • * + a p X p — a'X 

From (2-43), 

Pz = E(Z) = a> 

and 

<r\ = Var(Z) = a'Xa 


Moreover, by Result 4.2, Z has an N (a'/i, a'Xa) distribution. If a random sample 
Xi , X 2 ,..., X„ from the N p (p y X) population is available, a corresponding sample 
of Z’s can be created by taking linear combinations. Thus, 

Zj = a x Xj i + a 2 Xj 2 + ••• + a p X jp = a'X; j = 1,2,..., n 

The sample mean and variance of the observed values z x , z 2 , • .., z n are, by (3-36), 

z = a'x 
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S 2 = a'Sa 

where x and S are the sample mean vector and covariance matrix of the x/s, 

'^Simultaneous confidence intervals can be developed from a consideration of con¬ 
fidence intervals for a> for various choices of a. The argument proceeds as follows. 

For a fixed and cr| unknown, a 100( 1 - <*)% confidence interval for Mz - a p 
is based on student’s t-ratio 

_ z - mz _ V»(a'x ~ «'/*) (5-20) 

‘ sj Vn Va'Sa 


and leads to the statement 


t - u-Mi) 




i (a/2) -^- 


, V?Si 

tn-i( a / 2 ) \/n~ 


where r ,(«/2) is the upper 100(«/2)th percentile of a '-distribution with n - 1 d* 
Tneaualitv (5-21) can be interpreted as a statement about the components of the 
mea!ivector m For example,with a' = [l,0,...,0],aV = and (5-21) becomes 
the usual confidence interval for a normal population mean. (Note, in this case, that 
-L - *, ) Clearly, we could make several confidence statements about the com¬ 
ments of p, each with associated confidence coefficient 1 - a, by choosing differ¬ 
ent coefficient vectors a. However, the confidence associated with all of the 

State intuitively e |t would be desirable to associate a “collective” confidence coeffi¬ 
cient of 1 - a with the confidence intervals that can be generated by all choices of 
a However, a price must be paid for the convenience of a large simultaneous confi¬ 
dence coefficient: intervals that are wider (less precise) than the interval of (5-21) 

f ° r GiTen^a data set x t , x 2 ,--■, x„ and a particular a, the confidence interval in 
(5-21) is that set of a'/* values for which 


Vn (a'x - a'ft) 
I Va'Sa 


or, equivalently, 


n( a'x - a'ft) 2 n( a'(x - p)f 


< tl-iia/2) 


A simultaneous confidence region is given by the set of a'ft values such that t 2 is rel¬ 
atively small for all choices of a. It seems reasonable to expect that the constant 
f 2 _i(a/2) in (5-22) will be replaced by a larger value, c , when statements are devel¬ 
oped for many choices of a. 
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Considering the values of a for which t 2 ^ c 2 , we are naturally led to the deter¬ 
mination of 

n(a'(x - m)) 2 


Using the maximization lemma (2-50) with x = a, d = (x — /t ), and B = S, we get 


n(a'(x - n)) 


= n max - 


(a’(x - /*)) 


n(x — /t)'S _1 (x - p) = T 2 (5-23) 


with the maximum occurring for a proportional to S~*(x — /t). 

Result 5.3. Let X w X 2 ,..., X n be a random sample from an N p (p, £) population 
with % positive definite. Then, simultaneously for all a, the interval 


/ p(« - n 

n(n - p) 


P p .n-p{a) a'Sa, a'X + 


~ 1 ! -^.«-p(«)a , Sa 
n(n — p) 


will contain a'/t with probability 1 - a. 


Proof. From (5-23), 


T 2 = n(x-p)'S~\x-p)^c 2 implies a » < c 2 

a Sa 


for every a, or 


# _ /a'Sa 

ax_c V^r- aMSa c 


for every a. Choosing c 2 = p(n — 1 )F p n - p (a)/(n - p) [see (5-6)] gives intervals 
that will contain a'/i for all a, with probability 1 - a = P[T 2 < c 2 ]. ■ 


It is convenient to refer to the simultaneous intervals of Result 5.3 as 
T 2 ~intervals y since the coverage probability is determined by the distribution of T 2 . 
The successive choices a' = [1,0,...,0], a' = [0,1,...,0], and so on through 
a' = [0,0,..., 1] for the T 2 -intervals allow us to conclude that 



all hold simultaneously with confidence coefficient 1 — a. Note that, without modi¬ 
fying the coefficient 1 - a, we can make statements about the differences pi - p k 
corresponding to a' = [0,...,0, a k , 0,..., 0], where a t — 1 and 
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dk = - 1 . In this case a'Sa = - 2s,* + s**, and we have the statement 



The simultaneous T 2 confidence intervals are ideal for “data snooping.” The 
confidence coefficient 1 - a remains unchanged for any choice of a, so linear com¬ 
binations of the components m that merit inspection based upon an examination of 

the data can be estimated. . , 

In addition, according to the results in Supplement 5 A, we can include the state¬ 
ments about (//;, dk) belonging to the sample mean-centered ellipses 


n[Xi - Pi, x k - L'' 


*k ~ dk 


p(n - 1 ) 


Fnn-p(<x) (5-26) 


and still maintain the confidence coefficient (1 - a) for the whole set of statements. 

The simultaneous T 2 confidence intervals for the individual components of a 
mean vector are just the shadows, or projections, of the confidence ellipsoid on the 
component axes. This connection between the shadows of the ellipsoid and the si¬ 
multaneous confidence intervals given by (5-24) is illustrated in the next example. 


Example 5.4 (Simultaneous confidence intervals as shadows of the confidence ellipsoid) 

In Example 5.3, we obtained the 95% confidence ellipse for the means of the fourth 
roots of the door-closed and door-open microwave radiation measurements. The 95% 
simultaneous T 2 intervals for the two component means are, from (5-24), 






In Figure 5.2, we have redrawn the 95% confidence ellipse from Example 5.3. 
The 95% simultaneous intervals are shown as shadows, or projections, of this ellipse 
on the axes of the component means. ■ 


Example 5.5 (Constructing simultaneous confidence intervals and ellipses) The 

scores obtained by n = 87 college students on the College Level Examination Pro- 
eram (CLEP) subtest X x and the College Qualification Test (CQT) subtests X 2 and 
X are given in Table 5.2 on page 228 for X, = social science and history, 
X = verbal, and X 3 = science. These data give 
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Figure 5.2 Simultaneous T 2 -intervals for the component means as shadows of the 
confidence ellipse on the axes—microwave radiation data. 
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Let us compute the 95% simultaneous confidence intervals for and / 13 . 

We have 

P(» - 1) - , . 3(87-1) _ 3(86) 
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and we obtain the simultaneous confidence statements [see (5-24)] 
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or 

23.65 < /x 3 ^ 26.61 


With the possible exception of the verbal scores, the marginal Q-Q plots and two- 
dimensional scatter plots do not reveal any serious departures from normality for 
the college qualification test data. (See Exercise 5.18.) Moreover, the sample size is 
large enough to justify the methodology, even though the data are not quite normally 
distributed. (See Section 5.5.) 

The simultaneous r 2 -intervals above are wider than univariate intervals because 
all three must hold with 95% confidence. They may also be wider than necessary, be¬ 
cause, with the same confidence, we can make statements about differences. 

For instance, with a' = [0,1,-1], the interval for /x 2 - m 3 has endpoints 


(x 2 - *3) ± 


p(" - 1) p 
(n - p) Fp - n 


-p(-05) yj- 


s 22 + S33 " 2s 2 3 


= (54.69 - 25.13) db V09 


126.05 + 23.11 - 2(23.39) 


87 


= 29.56 ± 3.12 


so (26.44, 32.68) is a 95% confidence interval for p 3 . Simultaneous intervals 

can also be constructed for the other differences. 

Finally, we can construct confidence ellipses for pairs of means, and the same 
95% confidence holds. For example, for the pair (/x 2 , /x 3 ), we have 


87[54.69 - M2, 


25.13 - fi 3 \ 


"126.05 23.39" 

-1 

54.69 - M2 

_ 23.39 23.11_ 


25.13 - mb. 


= 0.849(54.69 - M2) 2 + 4.633(25.13 - M3) 2 

- 2 X 0.859(54.69 - M2) (25.13 - M3) =s 8.29 


This ellipse is shown in Figure 5.3 on page 230, along with the 95% confidence ellipses for 
the other two pairs of means. The projections or shadows of these ellipses on the axes are 
also indicated, and these projections are the r 2 -intervals. ■ 


A Comparison of Simultaneous Confidence Intervals 
with One-at-a-Time Intervals 


An alternative approach to the construction of confidence intervals is to consider 
the components Mi one at a time, as suggested by (5-21) with a' = [0, 
a i9 0,...,0] where a t = 1. This approach ignores the covariance structure of the 
p variables and leads to the intervals 


*1 - tn-M/2) yj^- < Ml s Xi + t n -i(a/2) yj^j- 

*2 ~ l n- 1(<*/2) =£ M2 — *2 + l n- 1(“/2) 

\ n \ n 


tn-Ml) 





< x 

- D 


+ 6.-iW2) 
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Vi 



50.5 54.5 58.5 

Figure 5.3 95% confidence ellipses for pairs of means and the simultaneous 
r 2 -intervals—college test data. 


Although prior to sampling, the ith interval has probability 1 - a of covering p,, 
we do not know what to assert, in general, about the probability of all intervals con¬ 
taining their respective p.’s. As we have pointed out, this probability is not 1 - a. 

To shed some light on the problem, consider the special case where the obser¬ 
vations have a joint normal distribution and 



Since the observations on the first variable are independent of those on the second 
variable, and so on, the product rule for independent events can be applied. Before 
the sample is selected, 

P[all r-intervals in (5-27) contain the p,’s] = (1 - a)(l - a) -(1 - a) 

= (1 - ay 

If 1 - a = .95 and p = 6, this probability is ( 95) 6 = .74. 
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To guarantee a probability oil — a that all of the statements about the compo¬ 
nent means hold simultaneously, the individual intervals must be wider than the sepa¬ 
rate f-intervals; just how much wider depends on both p and rc, as well a s on 1 - a. 

For 1 — « = .95, n = 15, and p = 4, the multipliers of \Zs it /n in (5-24) and 
(5-27) are 




and f„_ 1 (.025) = 2.145, respectively. Consequently, in this case the simultaneous in¬ 
tervals are 100(4.14 - 2.145)/2.145 = 93% wider than those derived from the one- 
at-a-time t method. 

Table 5.3 gives some critical distance multipliers for one-at-a-time f-intervals 
computed according to (5-21), as well as the corresponding simultaneous ^-inter¬ 
vals. In general, the width of the T 2 -intervals, relative to the f-intervals, increases for 
fixed n as p increases and decreases for fixed p as n increases. 



The comparison implied by Table 5.3 is a bit unfair, since the confidence level 
associated with any collection of T 2 -intervals, for fixed n and p, is .95, and the over¬ 
all confidence associated with a collection of individual t intervals, for the same n , 
can, as we have seen, be much less than .95. The one-at-a-time t intervals are too 
short to maintain an overall confidence level for separate statements about, say, all 
p means. Nevertheless, we sometimes look at them as the best possible information 
concerning a mean, if this is the only inference to be made. Moreover, if the one-at- 
a-time intervals are calculated only when the T 2 -test rejects the null hypothesis, 
some researchers think they may more accurately represent the information about 
the means than the T 2 -intervals do. 

The r 2 -intervals are too wide if they are applied only to the p component means. 
To see why, consider the confidence ellipse and the simultaneous intervals shown in 
Figure 5.2. If p, lies in its T 2 -interval and /x 2 lies in its T 2 -interval, then (p^, /a 2 ) lies in 
the rectangle formed by these two intervals. This rectangle contains the confidence 
ellipse and more.The confidence ellipse is smaller but has probability .95 of covering 
the mean vector /* with its component means and p 2 . Consequently, the probabil¬ 
ity of covering the two individual means pi and will be larger than .95 for the rec¬ 
tangle formed by the T 2 -intervals. This result leads us to consider a second approach 
to making multiple comparisons known as the Bonferroni method. 
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The Bonferroni Method of Multiple Comparisons 


Often, attention is restricted^mullanwIiK 6 intervau n of 
I" tb~ *«““»• » r line*combine^, 

Result 5.3. If the number pe simultaneous confidence intervals can be 

a> - aiMt + fl 2M2 p p n rec ise) than the simultaneous T -intervals, 

developed that are shor e J P jj is called the Bonferroni method , 
Hie alternative method for multlp '® “™ jn eaualky carrying that name, 
because it is developed r °m a P^ y confidence statements about m lin- 
Suppose that, prior to the ^ Cf de „ote a confidence state- 

ear combinationsa,M,»2M. • ■ ■ > a "’ M . 2* i = j _ a / = 1,2,...,m. Now (see 

ment about the value of a'iM with P[Ci true] 

Exercise 5.6), 

jp[aUC/ true] = 1 - P[atleast one C, false] ^ 


iii 

> 1 - 5) P(Q false) = 1 

f=i 

= 1 - (0£] + C*2 + " * + a w 


m 

2(1 — P(C,-true)) 

i=i 


t r,v a snecial case of the Bonferroni inequality, allows an investi¬ 
gator JcontrolTheoverall error rate 4ere flexibility of 

^important statements and balancing it by 

another choice for the less for the restricted set consisting 

I^t us develop smulan^ nformation 0 n the relative importance of these 
of the components Mi of M- f^ CK ing 1 . , 

components, we consider the individual t-interv 

- , , ( c ?l\ i = 1 , 2 ,...,m 

*/ ± 'tt-l^ 2 ) V n 

, C- ^ pry 4 - / ,(a/2m)VsJn contains Mi] = 1 “ a / m ’ 

with a,- = a/m. Since P[X, ± «-i l"/- 4 ' 

l — 1 f 2,..., m, we have, from (5-2o), 


r + * ( JL\ pi contains p x 

± t n -iy2m/\ n 


fa a cl 

■i, a11 ' S 1 " \ m + m + m 

J \ _, 


= 1 - a 


rn terms 


, 11 .«r,fm P nre level areater than or equal to 1 - a, we can 

Therefore, with an overall confidence tevei gr 

make the following rn = p statements: _ 

1 , -1.-(|) t/? s « s S| 4 '-'(5) $ ■ 

(5 - 29) 


■*****'MZ)'I ■? 
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The statements in (5-29) can be compared with those in (5-24). The percentage 
point t n -\(a/2p) replaces V(/i - \)pFp n ~ p {a)/(n ~ p), but otherwise the inter¬ 
vals are of the same structure. 

Example 5.6 (Constructing Bonferroni simultaneous confidence intervals and com¬ 
paring them with 7 2 -intervals) Let us return to the microwave oven radiation data 
in Examples 5.3 and 5.4. We shall obtain the simultaneous 95% Bonferroni confi¬ 
dence intervals for the means, il x and p, 2 , of the fourth roots of the door-closed and 
door-open measurements with a* = .05/2, i = 1,2. We make use of the results in 
Example 5.3, noting that n = 42 and r 41 (.05/2(2)) = r 41 (.0125) - 2.327, to get 

Xi ± r 4 i(.0125) = .564 ± 2.327 or .521 £ mi ^ -607 

x 2 ± r 41 (.0125) = .603 ± 2.327 or .560 s w < .646 

Figure 5.4 shows the 95% T 2 simultaneous confidence intervals for fr° m 

Figure 5.2, along with the corresponding 95% Bonferroni intervals. For each com¬ 
ponent mean, the Bonferroni interval falls within the r 2 -interval. Consequently, 
the rectangular (joint) region formed by the two Bonferroni intervals is contained 
in the rectangular region formed by the two T 2 -intervals. If we are interested only in 
the component means, the Bonferroni intervals provide more precise estimates than 


^2 



0.500 0.552 0.604 


Figure 5.4 The 95% T 2 and 95% Bonferroni simultaneous confidence intervals for the 
component means—microwave radiation data. 
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the r 2 -intervals. On the other hand, the 95% confidence region for p gives the 
plausible values for the pairs (p\ , P 2 ) when the correlation between the measured 
variables is taken into account. ^ 

The Bonferroni intervals for linear combinations a 'p and the analogous: 
r 2 -intervals (recall Result 5.3) have the same general form: 

/sTSa 

a'X ± (critical value) 

Consequently, in every instance where a, = a/m,. 

Length of Bonferroni interval = t n _ 1 (a/2m) ^ ^ 

Length of r 2 -interval lp(n - 1) ~ 

V~^r 

which does not depend on the random quantities X and S. As we have pointed out, for 
a small number m of specified parametric functions a'/*, the Bonferroni intervals will 
always be shorter. How much shorter is indicated in Table 5.4 for selected n and p. 

Table 5.4 (Length of Bonferroni Interval)/(Length of T 2 -Interval) 
for 1 - a = .95 and a t = .05/m 


m = p 

n 2 

4 

10 

15 .88 

.69 

.29 

25 .90 

.75 

.48 

50 .91 

.78 

.58 

100 .91 

.80 

.62 

00 .91 

.81 

.66 


We see from Table 5.4 that the Bonferroni method provides shorter intervals 
when m = p. Because they are easy to apply and provide the relatively short confi¬ 
dence intervals needed for inference, we will often apply simultaneous ^-intervals 
based on the Bonferroni method. 


5.5 Large Sample Inferences about a Population Mean Vector 

When the sample size is large, tests of hypotheses and confidence regions for p can 
be constructed without the assumption of a normal population. As illustrated in 
Exercises 5.15,516, and 5.17, for large n, we are able to make inferences about the 
population mean even though the parent distribution is discrete. In fact, serious de¬ 
partures from a normal population can be overcome by large sample sizes. Both 
tests of hypotheses and simultaneous confidence statements will then possess (ap¬ 
proximately) their nominal levels. 

The advantages associated with large samples may be partially offset by a loss in 
sample information caused by using only the summary statistics x, and S. On the 
other hand, since (x, S) is a sufficient summary for normal populations [see (4-21)], 
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the closer the underlying population is to multivariate normal, the more efficiently 
the sample information will be utilized in making inferences. 

All large-sample inferences about p are based on a ^-distribution. From (4-28), 
we know that (X — p)'(n~ l S) *(X - p) = n(X - p)' S _1 (X - p) is approxi¬ 
mately x 2 with p d.f., and thus, 

P[n(X - p)'S- l (X - /0 * X»1 = 1 - « (5-31) 

where * 2 (a) is the upper (100a)th percentile of the ^-distribution. 

Equation (5-31) immediately leads to large sample tests of hypotheses and simul¬ 
taneous confidence regions.These procedures are summarized in Results 5.4 and 5.5. 

Result 5.4. Let X x , X 2 , ...,X„bea random sample from a population with mean 
p and positive definite covariance matrix X. When n - p is large, the hypothesis 
H 0 : p = p 0 is rejected in favor of H Y : p ^ p 0 , at a level of significance approxi¬ 
mately a, if the observed 

«(* - Mo)'S _1 (x - Mo) > a£(“) 

Here ^(a) is the upper (100a)th percentile of a chi-square distribution with pd.f. ■ 

Comparing the test in Result 5.4 with the corresponding normal theory test in 
(5-7), we see that the test statistics have the same structure, but the critical values 
are different. A closer examination, however, reveals that both tests yield essential¬ 
ly the same result in situations where the ^ 2 -test of Result 5.4 is appropriate. This 
follows directly from the fact that (n - 1 )pF p ^ p (a)/{n - p) and x 2 p( a ) are a P“ 
proximately equal for n large relative to p . (See Tables 3 and 4 in the appendix.) 


Result 5.5. Let X 1 , X 2 ,..., X„ be a random sample from a population with mean 
p and positive definite covariance X. If n — p is large, 


a'X ± 




will contain a 'p, for every a, with probability approximately 1 - a. Consequently, 
we can make the 100(1 - a)% simultaneous confidence statements 



contains pi 
contains p 2 


contains p p 

and, in addition, for all pairs (p h p k ),i, k = 1, 2,..., p, the sample mean-centered 
ellipses 



n [*,• - Mi, x k - Mt] 


S ik S kk 


-1 


~ Mi 
X k ~ p k 


< X 2 p (a) contain (m,,M*) 
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Proof. The first part follows from Result 5A.1, with c 2 = x* P ( a )- The probability 
level is a consequence of (5-31). The statements for the p, are obtained by the spe¬ 
cial choices a' = [0,..., 0, o„ 0,..., a], where a t = 1, i = 1,2,..., p. The ellipsoids 
for pairs of means follow from Result 5A.2 with c 2 = The overall confidence 

level of approximately 1 - a for aU statements is, once again, a result of the large 
sample distribtttion theory summarized in (5-31). ■ 

The question of what is a large sample size is not easy to answer. In one or two 
dimensions, sample sizes in the range 30 to 50 can usually be considered large. As 
the number characteristics becomes large, certainly larger sample sizes are required 
for the asymptotic distributions to provide good approximations to the true distrib¬ 
utions of various test statistics. Lacking definitive studies, we simply state that n-p 
must be large and realize that the true case is more complicated. An application 
with p = 2 and sample size 50 is much different than an application with p = 52 and 
sample size 100 although both have n — p — 48. 

It is good statistical practice to subject these large sample inference procedures 
to the same checks required of the normal-theory methods. Although small to 
moderate departures from normality do not cause any difficulties for n large, 
extreme deviations could cause problems. Specifically, the true error rate may be far 
removed from the nominal level a. If, on the basis of Q-Q plots and other investiga¬ 
tive devices, outliers and other forms of extreme departures are indicated (see, for 
example, [2]), appropriate corrective actions, including transformations, are desir¬ 
able. Methods for testing mean vectors of symmetric multivariate distributions that 
are relatively insensitive to departures from normality are discussed in [11]. In some 
instances, Results 5.4 and 5.5 are useful only for very large samples. 

The next example allows us to illustrate the construction of large sample simul¬ 
taneous statements for all single mean components. 


Example 5.7 (Constructing large sample simultaneous confidence intervals) A music 
educator tested thousands of Finnish students on their native musical ability in order 
to set national norms in Finland. Summary statistics for part of the data set are given 
in Table 5.5. These statistics are based on a sample of n = 96 Finnish 12th graders. 


Table 5.5 Musical Aptitude Profile Means and Standard Deviations for 96 
12th-Grade Finnish Students Participating in a Standardization Program 


Variable Mean (x,) 

Xi = melody 28- 1 

X 2 - harmony 26.6 

X 3 = tempo 35.4 

X 4 = meter I 34.2 

X } = phrasing 23.6 

X 6 = balance 22.0 

X 7 = style 2 2.7 _ 

Source: Data courtesy of V. Sell. 


Raw score 


Standard deviation 
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Let us construct 90% simultaneous confidence intervals for the individual mean 
components i = 1, 2,...»7. 

From Result 5.5, simultaneous 90% confidence limits are given by 
Xj ± V*7(-10) i = 1,2,..., 7, where ^(.10) = 12.02. Thus, with approxi¬ 
mately 90% confidence, 


28.1 ± 1 

__5.76 

contains pi 

26.6 ± < 

V96 

contains /x 2 

35.4 ± ' 

__3.82 

VuM —= 

V96 

contains /x 3 

34.2 ± ^ 

,_5.12 

vTZ 02-^ 

V96 

,_3 76 

contains /x 4 

23.6 ± ” 

vH02 — 
V§6 

contains /x 5 

22.0 ± ' 

._3.93 

contains /x 6 

22.7 ± ^ 

V§6 

contains /x 7 


26.06 < Mi ss 30.14 
24.53 ^ /x 2 < 28.67 
34.05 < /x 3 < 36.75 
32.39 < /x 4 < 36.01 

22.27 < < 24.93 

20.61 < M6 ^ 23.39 

21.27 < /i 7 < 24.13 


Based, perhaps, upon thousands of American students, the investigator could hy¬ 
pothesize the musical aptitude profile to be 


Mo = [31,27,34,31,23,22,22] 

We see from the simultaneous statements above that the melody, tempo, and meter 
components of /*o do not appear to be plausible values for the corresponding means 
of Finnish scores. ■ 


When the sample size is large, the one-at-a-time confidence intervals for indi¬ 
vidual means are 



where z(a/2) is the upper 100(a/2)th percentile of the standard normal distribu¬ 
tion. The Bonferroni simultaneous confidence intervals for the m = p statements 
about the individual means take the same form, but use the modified percentile 
z(a/2p) to give 
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Table 5.6 gives the individual, Bonferroni, and chi-square-based (or shadow of 
the confidence ellipsoid) intervals for the musical aptitude data in Example 5.7. 


Table 5.6 The Large Sample 95% Individual, Bonferroni, and T 2 -Intervals for | 
the Musical Aptitude Data ___ | 


The one-at-a-time confidence intervals use z(.025) = 1.96. 

The simultaneous Bonferroni intervals use z(.025/7) = 2.69. 

The simultaneous T 2 , or shadows of the ellipsoid, use a^(.05) = 14.07. 


One-at-a-time 

Bonferroni Intervals 

Shadow of Ellipsoid 1 

Variable 

Lower 

Upper 

Lower 

Upper 

Lower 

Upper 

X x ~ melody 

26.95 

29.25 

26.52 

29.68 

25.90 

30.30 

X 2 ~ harmony 

25.43 

27.77 

24.99 

28.21 

24.36 

28.84 

Xi = tempo 

34.64 

36.16 

34.35 

36.45 

33.94 

36.86 

X 4 = meter 

33.18 

35.22 

32.79 

35.61 

32.24 

36.16 

X 5 = phrasing 

22.85 

24.35 

22.57 

24.63 

22.16 

25.04 

Xa - balance 

21.21 

22.79 

20.92 

23.08 

20.50 

23.50 

X-i = style 

21.89 

23.51 

21.59 

23.81 

21.16 

24.24 


Although the sample size may be large, some statisticians prefer to retain the 
F- and r-based percentiles rather than use the chi-square or standard normal-based 
percentiles. The latter constants are the infinite sample size limits of the former 
constants. The F and t percentiles produce larger intervals and, hence, are more con¬ 
servative. Table 5.7 gives the individual, Bonferroni, and F-based, or shadow of the 
confidence ellipsoid, intervals for the musical aptitude data. Comparing Table 5.7 
with Table 5.6, we see that all of the intervals in Table 5.7 are larger. However, with 
the relatively large sample size n = 96, the differences are typically in the third, or 
tenths, digit. 



* 



5.6 


Table 5.7 The 95% Individual, Bonferroni, and T 2 -Intervals for the 

Musical Aptitude Data 

The one-at-a-time confidence intervals use r 95 (.025) = 1.99. 


The simultaneous Bonferroni intervals use r 95 (.025/7) = 2.75. 


The simultaneous T 2 , or shadows of the ellipsoid, 

use F 7 89 (.05) = 2.11. 



One-at-a-time 

Bonferroni Intervals 

Shadow of Ellipsoid 

Variable 

Lower 

Upper 

Lower 

Upper 

Lower 

Upper 

Xi = melody 

26.93 

29.27 

26.48 

29.72 

25.76 

30,44 

X 2 - harmony 

25.41 

27.79 

24.96 

28.24 

24.23 

28.97 

X 3 = tempo 

34.63 

36.17 

34.33 

36.47 

33.85 

36.95 

X 4 = meter 

33.16 

35.24 

32.76 

35.64 

32.12 

36.28 

X 5 = phrasing 

22.84 

24.36 

22.54 

24.66 

22.07 

25.13 

X 6 = balance 

21.20 

22.80 

20.90 

23.10 

20.41 

23.59 

X 7 = style 

21.88 

23.52 

21.57 

23.83 

21.07 

24.33 


Multivariate Quality Control Charts 

To improve the quality of goods and services, data need to be examined for causes 
of variation. When a manufacturing process is continuously producing items or 
when we are monitoring activities of a service, data should be collected to evaluate 
the capabilities and stability of the process. When a process is stable, the variation is 
produced by common causes that are always present, and no one cause is a major 
source of variation. 

The purpose of any control chart is to identify occurrences of special causes of 
variation that come from outside of the usual process. These causes of variation 
often indicate a need for a timely repair, but they can also suggest improvements to 
the process. Control charts make the variation visible and allow one to distinguish 
common from special causes of variation. 

A control chart typically consists of data plotted in time order and horizontal 
lines, called control limits, that indicate the amount of variation due to common 
causes. One useful control chart is the X -chart (read AT-bar chart). To create an 
X -chart, 

1. Plot the individual observations or sample means in time order. 

2. Create and plot the centerline x , the sample mean of all of the observations. 

3. Calculate and plot the control limits given by 

Upper control limit (UCL) = x + 3(standard deviation) 

Lower control limit (LCL) = x — 3(standard deviation) 

The standard deviation in the control limits is the estimated standard deviation 
of the observations being plotted. For single observations, it is often the sample 
standard deviation. If the means of subsamples of size m are plotted, then 
the standard deviation is the sample standard deviation divided by \frn. The 
control limits of plus and minus three standard deviations are chosen so that 
there is a very small chance, assuming normally distributed data, of falsely signal¬ 
ing an out-of-control observation—that is, an observation suggesting a special 
cause of variation. 


Example 5.8 (Creating a univariate control chart) The Madison, Wisconsin, police 
department regularly monitors many of its activities as part of an ongoing quality 
improvement program. Table 5.8 gives the data on five different kinds of over¬ 
time hours. Each observation represents a total for 12 pay periods, or about half 
a year. 

We examine the stability of the legal appearances overtime hours. A computer 
calculation gives x x = 3558. Since individual values will be plotted, x x is the same as 
x v Also, the sample standard deviation is ~ 607, and the control limits are 

UCL = x x + 3(V57i) = 3558 + 3(607) = 5379 

LCL = x Y - 3( VJ^i) = 3558 - 3(607) = 1737 
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The legal appearances overtime hours are stable over the period in which the 
data were collected. The variation in overtime hours appears to be due to common 
causes, so no special-cause variation is indicated. m 

With more than one important characteristic, a multivariate approach should be 
used to monitor process stability. Such an approach can account for correlations 
between characteristics and will control the overall probability of falsely signaling a 
special cause of variation when one is not present. High correlations among the 
variables can make it impossible to assess the overall error rate that is implied by a 
large number of univariate charts. 

The two most common multivariate charts are (i) the ellipse format chart and 
(ii) the T 2 -chart. 

Two cases that arise in practice need to be treated differently: 

1. Monitoring the stability of a given sample of multivariate observations 

2. Setting a control region for future observations 

Initially, we consider the use of multivariate control procedures for a sample of mul¬ 
tivariate observations Xj, x 2 ,..., x„. Later, we discuss these procedures when the 
observations are subgroup means. 

Charts for Monitoring a Sample of Individual Multivariate 
Observations for Stability 

We assume that X u X 2 ,..., X„ are independently distributed as N p (fx, X). By 
Result 4.8, 



has 

E(Xj - X) = 0 = (1 - /TV - (« - 1 )«v 

and 

Cov(X y - X) = (l - + (n - l)rc~ 2 £ = X 

Each X j — X has a normal distribution but,X y - X is not independent of the sam¬ 
ple covariance matrix S. However to set control limits, we approximate that 
(Xy — X)'S -1 (Xy - X) has a chi-square distribution. 

Ellipse Format Chart. The ellipse format chart for a bivariate control region is the 
more intuitive of the charts, but its approach is limited to two variables. The two 
characteristics on the jth unit are plotted as a pair (x yl , Jt y2 ). The 95% quality ellipse 
consists of all x that satisfy 


(x - x)'S _1 (x - x) =s *2(.05) 


(5-32) 
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Example 5.9 (An ellipse format chart for overtime hours) Let us refer to Example 
5.8 and create a quality ellipse for the pair of overtime characteristics (legal appear¬ 
ances, extraordinary event) hours. A computer calculation gives 



and S — 


367,884.7 -72,093.8~| 

-72,093.8 1,399,053.1 J 


We illustrate the quality ellipse format chart using the 99% ellipse, which con¬ 
sists of all x that satisfy 

(x - x)'S-‘(x - I) s xl(.01) 

Here p - 2, so ^(.01) = 9.21, ancl t ^ e ellipse becomes 

Sll s 22 1 te-**) 2 ) 

- > 

(367844.7 x 1399053.1) 

= 367844.7 X 1399053.1 - (-72093.8) 2 


(X! - 3558) 2 
367844.7 


2(-72093.8) 


(xi - 3558) (x 2 - 1478) 
367844.7 x 1399053.1 


+ 


(x 2 -1478) 2 \ 
1399053.1 } 


<9.21 


This ellipse format chart is graphed, along with the pairs of data, in Figure 5.6, 




o 



1500 2500 3500 4500 5500 

Appearances Overtime 


Figure 5.6 The quality control 
99% ellipse for legal 
appearances and extraordinary 
event overtime. 
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Extraordinary Event Hours 



Observation Number 

Figure 5.7 The2f -chart for x 2 — extraordinary event hours. 

Notice that one point, indicated with an arrow, is definitely outside of the el¬ 
lipse. When a point is out of the control region, individual^ charts are constructed. 
TheX -chart for x x was given in Figure 5.5; that for x 2 is given in Figure 5.7. 

When the lower control limit is less than zero for data that must be non¬ 
negative, it is generally set to zero. The LCL = 0 limit is shown by the dashed line in 
Figure 5.7. 

Was there a special cause of the single point for extraordinary event overtime 
that is outside the upper control limit in Figure 5.7? During this period, the United 
States bombed a foreign capital, and students at Madison were protesting. A major¬ 
ity of the extraordinary overtime was used in that four-week period. Although, by its 
very definition, extraordinary overtime occurs only when special events occur and is 
therefore unpredictable, it still has a certain stability. ■ 

r 2 -Chart. A T 2 -chart can be applied to a large number of characteristics. Unlike the 
ellipse format, it is not limited to two variables. Moreover, the points are displayed in 
time order rather than as a scatter plot, and this makes patterns and trends visible. 
For the Jth point, we calculate the T 2 -statistic 

Tj = (\j - x)'S _ 1 (x / - x) (5-33) 

We then plot the revalues on a time axis. The lower control limit is zero, and we use 
the upper control limit 

UCL = * 2 (.05) 

or, sometimes, ^(.01). 

There is no centerline in the T 2 -chart. Notice that the T 2 -statistic is the same as 
the quantity d 2 used to test normality in Section 4.6. 
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- mn |„ 5 I o /A T J -ehart for overtime hours) Using the police department data i n j 

J* 3 mnle S R we construct a T 2 -plot based on the two variables X, - legal appear-* 
Example 5.8, we co p event hours. T 2 -charts with more than tw<| 

526. We - . - -01 » be . ** 

^er’-ctoT m ’that (he pair (legal appearances amort.' 

SfiiattS value of ertr.ordinary even, overt,^ 

during that period. 

12 H 


10 


6 H 


4 H 


2 H 


10 


t2 


I 

14 


"T" 

16 


Period 


FiguK 5.8 The T^-dtart (or legal appearances bouts and emaordinap. event hours o - 01. 

When the ("dfb'^b^^^'^^^j^^o'^AnKuhfied'regroniMSedon'Bon^roih 
^rX""i P pun«»=:-n.e *th variable h out o, control U * 

does not lie in the interval ,— 

(x* - r„-i(.005/p)V^, ** + t n -i(.005/p)Vs**) 

where p is the total number of measured variable* 

inputs to Ihe^autorrtatedwddrng 'twch ^ good quality. In order to con- 

ttoUlteprowss one prtMesstmginee, measured four crirical variables: 

AT, = Voltage (volts) 

X 2 = Current (amps) 

X 3 = Feed speed(in/min) 

/♦._a \ /"I flsxur /rfm'i 



Table 5.9 gives the values of these variables at five-second intervals. 


Table 5.9 Welder Data 

Case 

Voltage (^0 

Current (X 2 ) 

Feed speed (Jf 3 ) 

Gas flow (X 4 ) 

1 

23.0 

276 

289.6 

51.0 

2 

22.0 

281 

289.0 

51.7 

3 

22.8 

270 

288.2 

51.3 

4 

22.1 

278 

288.0 

52.3 

5 

22.5 

275 

288.0 

53.0 

6 

22.2 

273 

288.0 

51.0 

7 

22.0 

275 

290.0 

53.0 

8 

22.1 

268 

289.0 

54.0 

9 

22.5 

277 

289.0 

52.0 

10 

22.5 

278 

289.0 

52.0 

11 

22.3 

269 

287.0 

54.0 

12 

21.8 

274 

287.6 

52.0 

13 

22.3 

270 

288.4 

51.0 

14 

22.2 

273 

290.2 

51.3 

15 

22.1 

274 

286.0 

51.0 

16 

22.1 

in 

287.0 

52.0 

17 

21.8 

277 

287.0 

51.0 

18 

22.6 

276 

290.0 

51.0 

19 

22.3 

278 

287.0 

51.7 

20 

23.0 

266 

289.1 

51.0 

21 

22.9 

271 

288.3 

51.0 

22 

21.3 

274 

289.0 

52.0 

23 

21.8 

280 

290.0 

52.0 

24 

22.0 

268 

288.3 

51.0 

25 

22.8 

269 

288.7 

52.0 

26 

22.0 

264 

290.0 

51.0 

27 

22.5 

273 

288.6 

52.0 

28 

22.2 

269 

288.2 

52.0 

29 

22.6 

273 

286.0 

52.0 

30 

21.7 

283 

290.0 

52.7 

31 

21.9 

273 

288.7 

55.3 

32 

22.3 

264 

287.0 

52.0 

33 

22.2 

263 

288.0 

52.0 

34 

22.3 

266 

288.6 

51.7 

35 

22.0 

263 

288.0 

51.7 

36 

22.8 

272 

289.0 

52.3 

37 

22.0 

277 

287.7 

53.3 

38 

22.7 

272 

289.0 

52.0 

39 

22.6 

274 

287.2 

52.7 

40 

22.7 

270 

290.0 

51.0 

Source: Data courtesy of Mark Abbotoy. 
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The normal assumption is reasonable for most variables, but we take the natur- 
al logarithm of gas flow. In addition, there is no appreciable serial correlation for 
successive observations on each variable. 

A T 2 -c hart for the four welding variables is given in Figure 5.9. The dotted line 
is the 95% limit and the solid line is the 99% limit. Using the 99% limit, no points* 
are out of control, but case 31 is outside the 95% limit. 

What do the quality control ellipses (elfipse format charts) show for two vari¬ 
ables? Most of the variables are in control. However, the 99% quality ellipse for gas 
flow and voltage, shown in Figure 5.10, reveals that case 31 is out of control and 
this is due to an unusually large volume of gas flow. The univariate X chart for 
ln(gas flow), in Figure 5.11, shows that this point is outside the three sigma limits. 
It appears that gas flow was reset at the target for case 32. All the other univariate 
X -charts have all points within their three sigma control limits. 




Voltage 


Figure 5.10 The 99% quality 
control ellipse for ln(gas flow) and 
voltage. 
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JU T_ _ __ , * 

Figure 5.11 The urn variate 

Case x -chart for ln(gas flow). 


In this example, a shift in a single variable was masked with 99% limits, or almost 
masked (with 95% limits), by being combined into a single T 2 -value. ■ 


Control Regions for Future Individual Observations 

The goal now is to use data x l9 x 2 ,..., x„, collected when a process is stable, to set a 
control region for a future observation x or future observations. The region in which 
a future observation is expected to lie is called a forecast, or prediction, region . If the 
process is stable, we take the observations to be independently distributed as 
N p (n, X). Because these regions are of more general importance than just for mon¬ 
itoring quality, we give the basic distribution theory as Result 5.6. 


Result 5.6. Let Xj, X 2 , ..., X„ be independently distributed as N p (fi, X), and let 
X be a future observation from the same distribution.Then 

T 2 = — (X - X)'S -1 (X - X) is distributed as — —p 

n + 1 n - p p ' p 

and a 100(1 — a)% p-dimensional prediction ellipsoid is given by all x satisfying 


(x - x)'S _1 (x - x) 


(" 2 ~ 1)P 
n(n - p) 


Fp,n-p{oi) 


Proof. We first note that X — X has mean 0. Since X is a future observation, X and 
X are independent, so 

Cov(X - X) = Cov(X) + Cov(X) = X + -% = — + 2 

n n 

and, by Result 4.8, Vn/(n + 1) (X - X) is distributed as N p ( 0,X). Now, 
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which combines a multivariate normal, N p ( 0, X), random vector and an independent 
Wishart, W p rt _ 1 (2), random matrix in the form 

/multivariate normal V / Wishart random matrix \ 1 /multivariate normal^ 

\ random vector / \ d.f. / \ random vector ) 

has the scaled T distribution claimed according to (5-8) and the discussion on 
page 213. 

The constant for the ellipsoid follows from (5-6). m 

Note that the prediction region in Result 5.6 for a future observed value x is an 
ellipsoid. It is centered at the initial sample mean x, and its axes are determined by 
the eigenvectors of S. Since 

_ „ (n 2 - l)p 

P (X - X)S~ (X - X) < _ ' p) />„-,(«) — \ — a 

before any new observations are taken, the probability that X will fall in the predic¬ 
tion ellipse is 1 - a. 

Keep in mind that the current observations must be stable before they can be 
used to determine control regions for future observations. 

Based on Result 5.6, we obtain the two charts for future observations. 

Control Ellipse for Future Observations 

With p - 2, the 95% prediction ellipse in Result 5.6 specializes to 
_ _ (n 2 — 1)2 

(x - x) S *(x - x) < — _ — F 2> „- 2 (.05) (5-34) 

Any future observation x is declared to be out of control if it falls out of the con¬ 
trol ellipse. 

Example 5.12 (A control ellipse for future overtime hours) In Example 5.9, we 
checked the stability of legal appearances and extraordinary event overtime hours. 
Let’s use these data to determine a control region for future pairs of values. 

From Example 5.9 and Figure 5.6, we find that the pair of values for period 11 
were out of control. We removed this point and determined the new 99% ellipse. All 
of the points are then in control, so they can serve to determine the 95% prediction 
region just defined for p = 2. This control ellipse is shown in Figure 5.12 along with 
the initial 15 stable observations. 

Any future observation falling in the ellipse is regarded as stable or in control. 
An observation outside of the ellipse represents a potential out-of-control observa¬ 
tion or special-cause variation. ■ 

7 2 -Chart for Future Observations 

For each new observation x, plot 

7-2 = 7+T (x - " *) 


Extraordinary Event Overtime 
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Appearances Overtime 


Figure 5.12 The 95% control 
ellipse for future legal 
appearances and extraordinary 
event overtime. 


in time order. Set LCL = 0, and take 

Points above the upper control limit represent potential special cause variation 
and suggest that the process in question should be examined to determine 
whether immediate corrective action is warranted. See [9] for discussion of other 
procedures. 


Control Charts Based on Subsample Means 

It is assumed that each random vector of observations from the process is indepen¬ 
dently distributed as N p ( 0 , X). We proceed differently when the sampling procedure 
specifies that m > 1 units be selected, at the same time, from the process. From the 
first sample, we determine its sample mean X x and covariance matrix S x . When 
the population is normal, these two random quantities are independent. 

For a general subsample mean X ; , X ; — X has a normal distribution with 
mean 0 and 


Cov (Xy - X) = 



\ Cov (Xj) 


+ 


—-^2"—Cov(Xj) 



nm 
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where i 

_ 1 " _ 

x -»s x ' : 

As will be described in Section 6.4, the sample covariances from the n sub-~ 
samples can be combined to give a single estimate (called Spo 0 , ed in Chapter 6) of the 
common covariance X. This pooled estimate is : 

S = -(S 1 + S 2 + --- + S„) j 

n 

Here (nm — n )S is independent of each X ; and, therefore, of their mean X. 
Further, (nm - n)S is distributed as a Wishart random matrix with nm - n degrees 
of freedom. Notice that we are estimating X internally from the data collected in 
any given period. These estimators are combined to give a single estimator with a 
large number of degrees of freedom. Consequently, 

T 2 = J 1 "L. (X, - X)'S-\X, - X) (5-35) 

n - 1 

is distributed as 

(nm - n)p 

” I 1 \ r p,nm-n~p +1 

(nm - n - p + 1) 

Ellipse Format Chart. In an analogous fashion to our discussion on individual 
multivariate observations, the ellipse format chart for pairs of subsample means is 

<« - «>'*"<* - *> « ^—■ ( - 05 > (MS) 

although the right-hand side is usually approximated as *2(-05)/m. 

Subsamples corresponding to points outside of the control ellipse should be 
carefully checked for changes in the behavior of the quality characteristics being 
measured. The interested reader is referred to [10] for additional discussion. 

J 2 -Chart. To construct a T 2 -chart with subsample data and p characteristics, we 
plot the quantity 

T) = m(Xj - XyS-'&j - X) 

for j ~ 1,2,..., n, where the 

(n - l)(m - 1 )p r f 

UCL — ~ ' , i \ Fp,nm-n-p+l('Q ) 

(nm - n - p + 1) 

The UCL is often approximated as x* P ( 05.) when n is large. 

Values of T) that exceed the UCL correspond to potentially out-of-control or 
special cause variation, which should be checked. (See [10].) 
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Control Regions for Future Subsample Observations 

Once data are collected from the stable operation of a process, they can be used to 
set control limits for future observed subsamplejneans. 

If X is a future subsample mean, then X — X has a multivariate normal distrib¬ 
ution with mean 0 and 

_ _ 1 _ (n + 1) 

Cov(X - X) = Cov(X) + — Cov(Xi) = --^X 

n nm 

Consequently, 

is distributed as 

(nm — n)p 
(nm — n - p + 1) 


Control Ellipse for Future Subsample Means. The prediction ellipse for a future 
subsample mean for p = 2 characteristics is defined by the set of all x such that 


(x - x)'S *(x - x) -- — -^■F2,«m-''-l(-05) ( 5 ' 3? ) 

m(nm — n - 1) 

where, again, the right-hand side is usually approximated as *§(.05)/m. 


J 2 -Chart for Future Subsample Means. As before, we bring n/(n + 1) into the 
control limit and plot the quantity 

T 2 = m(X - X)'S _1 (X - X) 


for future sample means in chronological order. The upper control limit is then 


UCL 


(n + l)(m - 1 )p 
(nm-n- p + 1) 


(.05) 


The UCL is often approximated as * 2 (.05) when n is large. 

Points outside of the prediction ellipse or above the UCL suggest that the cur¬ 
rent values of the quality characteristics are different in some way from those of the 
previous stable process. This may be good or bad, but almost certainly warrants a 
careful search for the reasons for the change. 


Inferences about Mean Vectors When Some 
Observations Are Missing 

Often, some components of a vector observation are unavailable. This may occur be¬ 
cause of a breakdown in the recording equipment or because of the unwillingness of 
a respondent to answer a particular item on a survey questionnaire. The best way to 
handle incomplete observations, or missing values, depends, to a large extent, on the 
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experimental context. If the pattern of missing values is closely tied to the value of 
the response, such as people with extremely high incomes who refuse to respond in a 
survey on salaries, subsequent inferences may be seriously biased. To date, no statisti¬ 
cal techniques have been developed for these cases. However, we are able to treat sit¬ 
uations where data are missing at random—that is, cases in which the chance 
mechanism responsible for the missing values is not influenced by the value of the 
variables. 

A general approach for computing maximum likelihood estimates from incom¬ 
plete data is given by Dempster, Laird, and Rubin [5]. Their technique, called the 
EM algorithm , consists of an iterative calculation involving two steps. We call them 
the prediction and estimation steps: 

1. Prediction step . Given some estimate 0 of the unknown parameters, predict 
the contribution of any missing observation to the (complete-data) sufficient 
statistics. 

2. Estimation step . Use the predicted sufficient statistics to compute a revised 
estimate of the parameters. 

The calculation cycles from one step to the other, until the revised estimates do 
not differ appreciably from the estimate obtained in the previous iteration. 

When the observations Xj, X 2 ,... , X„ are a random sample from a p-variate 
normal population, the prediction-estimation algorithm is based on the complete- 
data sufficient statistics [see (4-21)] 

Ti - 2 X; = nX 

/=! 

and 

T 2 = i X,X- = (n - 1)S + nXX 1 
i =i 

In this case, the algorithm proceeds as follows: We assume that the population mean 
and variance —fi and X, respectively—are unknown and must be estimated. 


Prediction step. For each vector x. with missing values, let x.- denote the miss- 

(2) 7 1 

ing components and x} denote those components which are available. Thus, 

*: = [*/ 1) \xj 2) ']- 

Given estimates /£ and X from the estimation step, use the mean of the condi¬ 
tional normal distribution of x (1 \ given x (2 \ to estimate the missing values. That is, 1 

xf = E(X j 1} I xj 2) ; £, X) = m (1) + X 12 XH(xj 2) - £ (2) ) (5-38) 

estimates .the contribution of xj 1 ^ to Tj. 

Next, the predicted contribution of xj 1 ^ to T 2 is 


xj l) x< 1} ' = E(X) l) Xy 


; X) = X n - .Xi 2 X 2 2^2i + x) 


*If all the components Xj are missing, set Xj = J1 and xjSj - X + £/T. 


(5-39) 
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= £(X} 1 >xf»'|xf ) ; £,X) = *}» x < 2 »' 


Esttmatto. step, Depute ,h. reused ^ 

tl = ^' ^ = (5-40) 

ta ‘ he of Ihe prediction-estimation algomhm 







o 

2 

1 



We obtain the initial smnpk'a WectoTS *> and x < are missing. 

4 


; h „s:r rase * ,or 

construct these estimates using the divisor L b “variance estimates. We shall 
duces the maximum likelihoocfestimate 2 Thus^ 1186 ^ al80nthm eventuall y P r °- 

df n = ( ^ 2 jtiIrr 6 Li_( 5 _ r 6) 2 + ( 6 6 ) 2 j 

4 "~ = o 

~ _ 1 _ 5 2 

^22 - j > *33 = J 

^12 = -^- — + — 7 ~ 6 ^ 2 ~ 1 } + ( 5 ~ 6)(1 - 1 ) + (6 - 6)(1 - l) 

= 1 4 
4 
3 

°23 - a 13 = 1 

contributions of the mTs^nTvalues'to^he 1116 ^ 1 * 131 eStlmates & and 2 to predict the 
and (5-39).] S “ t0 the sufficlent Tj and T 2 . (See (5-38) 
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The first component of x, is missing, so we partition jl and X as 


! CTi2 

<?13 


Sn 

Sl 2 

t &22 

0"23 

= 



i ~ ~ 


2 2 i 

x 22 

i u- 23 

0-33 J 



and predict £ 

n ri 11-1 fo — 1 1 J 

x\2~ ^2 = 6 + [1 il 2 * , , =5.73 | 

JT]3 - M3J ^ . J Li |J L 3 " 4 -l I 

~i 3n_! r i *1 -1 

"T ~ V v-iv + T 2 = i - f 1 ll 2 * j + (5.73) 2 = 32.99 

x\i = 0*11 — Xi 2 S 22 Z21 + -^11 2 [4’ J 3 5 2 

L4 2— 1 U- — 1 ’ 

i^T^]=5n[^. x 13 ]=5.73[0, 3] = [0, 17.18] 

For the two missing components of x 4 , we partition n and Z as 

'Mil r S d)-i ~ r?“ ~ 3 1 _ [Ml 12 . 

M = M2 I = ~(2f > X ~-- 2 -x ? - ? -{--S ? - 3 - Zo.l z« 


and predict 

^43 = 5;m,z)= ! 

_-*-42 _| \L* 42 J ' 

“[i_ + j_®" I( 5 - 4 )= _i. 

for the contribution toTj. Also, from (5-39), 


*41 *41*42 _ ( Xh X41X42 

X l * 41*42 *42 


+ Xi 2 X 22 (*43 ^3) 

^ 2 


j_-^41-^42 *42 


*43 =» 5;m,Z 


-] ll-IYIer'i 

U 2J L-4J 


1 f] + ”3 [6.4 1.3] 


41.06 8.27 
8.27 1.97 




64 (5)- 320 
13 11 « 
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are the contributions to T 2 . Thus, the predicted complete-data sufficient statistics 
are 


|*11 + *21 + *31 + *41 

*71*12 + *21*22 + *31*32 + *77*42 x\ 2 + x\ 2 + x\ 2 + x\ 2 

*77*13 + *21*23 + *31*33 + *41*43 *12*13 +*22*23 + *32*33 +*42*43 *13+*23+*33 +*43 


31 + *41 


32.99 + 7 2 + 5 2 + 41.06 
0 + 7(2) + 5(1) + 8.27 

17.18 + 7(6) + 5(2) + 32 

148.05 27.27 101.18 ~ 

27.27 6.97 20.50 

101.18 20.50 74.00 


0 2 + 2 2 + l 2 + 1.97 
0(3) + 2(6) + 1(2) + 6.5 


3 2 + 6 2 + 2 2 + 5‘ 


This completes one prediction step. 

The next estimation step, using (5-40), provides the revised estimates 2 


M=- t i 


Z = -T 2 - mm' 


[6.03 1.08 4.00] 


Note that a n — .61 and 3 ; 22 = -59 are larger than the corresponding initial esti¬ 
mates obtained by replacing the missing observations on the first and second vari¬ 
ables by the sample means of the remaining values. The third variance estimate cr 33 
remains unchanged, because it is not affected by the missing components. 

The iteration between the prediction and estimation steps continues until the 
elements of Jl and X remain essentially unchanged. Calculations of this sort are 
easily handled with a computer. ■ 


2 The final entries in £ are exact to two decimal places. 
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Once final estimates A and £ are obtained and relatively few missing compo¬ 
nents occur in X, it seems reasonable to treat 

all such that n( A - ~ m) - X 2 p( a ) ( 5 ‘41) 

as an annroximate 100(1 - «)% confidence elUpsoid.The simultaneous confidence 
statements^would then follow as in Section 5.5, but with 5 replaced by m and S re- 
placed by X. 

Caution. The prediction-estimation algorithm we discussed is developed on the 
basis that component observations are missing at random. If missing va ues are re¬ 
lated to the response levels, then handling the missing values as suggested may in- 
troduce serious biases into the estimation procedures. Typically, missing values are 
related to the responses being measured. Consequently we must be dubious of any 
computational scheme that fills in values as if they were lost at random When more 
thana few values are missing, it is imperative that the investigator search for the sys- 
tematic causes that created them. 


Difficulties Due to Time Dependence in Multivariate 
Observations 

For the methods described in this chapter, we have assumed that the multivariate 
observations X, X 2 ,..., X„ constitute a random sample; that is, they are indepen¬ 
dent of one another. If the observations are collected over time, this assumption 
may not be valid. The presence of even a moderate amount of time dependence 
among the observations can cause serious difficulties for tests, confidence regions, 
and simultaneous confidence intervals, which are all constructed assuming that in 

deP WewM Ulusuate the nature of the difficulty when the t ™ e J e P® nde "“ 
represented as a multivariate first order autoregressive [AR(1)] model. Let the 
p x 1 random vector X, follow the multivariate AR(1) model 

X, - ft = 4>(X,-1 - M) + e, (5-42) 

where the e, are independent and identically distributed with E[e,] =» and 
Cov(e r ) = and all of the eigenvalues of the coefficient matrix <P are between 1 

and 1 . Under this model Cov(X„X,- r ) = <*> r £, where 

00 

1=0 

The AR(1) model (5-42) relates the observation at time t, to the observation at time 
t- 1 through the coefficient matrix 4>. Further, the autoregressive model says he 
observations are independent, under multivariate normality if aU the entries m the 
coefficient matrix * are O.Tbe name autoregressive model comes from theTact that 
(5-42) looks like a multivariate version of a regression with X, as the dependent 
variable and the previous value X,-, as the independent variable. 
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As shown in Johnson and Langeland [8], 

S—S = —£ (X, - X)(X, -X)'—*2 X 
n 1 / = 1 

where the arrow above indicates convergence in probability, and 

Cov(n -1/2 2 X,) —* (I — Or'Sx + £x(I - t')' 1 - 2 X (5-43) 

Moreover, for large n , Vn (X — ft) is approximately normal with mean 0 and covari¬ 
ance matrix given by (5-43). 

To make the calculations easy, suppose the underlying process has <I> = <t>\ 
where \4> \ < 1. Now consider the large sample nominal 95% confidence ellipsoid 
for ft. 

{all ft such that /i(X - /i)'S -1 (X - ft) < ^(.05)} 

This ellipsoid has large sample coverage probability .95 if the observations are inde¬ 
pendent. If the observations are related by our autoregressive model, however, this 
ellipsoid has large sample coverage probability 

p[x 2 P s a - *)(i + <Ar'4(- 05 )] 

Table 5.10 shows how the coverage probability is related to the coefficient 4> and the 
number of variables p. 

According to Table 5.10, the coverage probability can drop very low, to .632, 
even for the bivariate case. 

The independence assumption is crucial, and the results based on this assump¬ 
tion can be very misleading if the observations are, in fact, dependent. 


Table 5.10 Coverage Probability of the Nominal 95% Confidence 
Ellipsoid 


-.25 

0 

4> 

.25 

.5 

1 

.989 

.950 

.871 

.742 

2 

.993 

.950 

.834 

.632 

P 5 

.998 

.950 

.751 

.405 

10 

.999 

.950 

.641 

.193 

15 

1.000 

.950 

.548 

.090 





Simultaneous Confidence 
Intervals and Ellipses as Shadows 


of the ^-Dimensional Ellipsoids 


We begin this supplementary section by establishing the general result concerning 
the projection (shadow) of an ellipsoid onto a line. 

Result 5A. 1. Let the constant c > 0 and positive definite pxp matrix A deter¬ 
mine the ellipsoid {z: z'A _1 z < c 2 }. For a given vector u # 0, and z belonging to the 
ellipsoid, the 

Projection (shadow) of 
{z f A“ l z < c 2 } onu 

which extends from 0 along u with length c\/u' Au/u'u. When u is a unit vector, the 
sha dow e xtends cVu'Au units, so |z'u| < cVu'Au. The shadow also extends 
cVu'Au units in the -u direction. 

Proof. By Definition 2A.12, the projection of any z on u is given by (z'u) u/iT u. Its 
squared length is (z'u) /u'u. We want to maximize this shadow over all z with 
z'A~ l z ^ c 2 . The extended Cauchy-Schwarz inequality in (2-49) states that 
(b'd) 2 < (b'Bd) (d'B _l d), with equality when b = A;B" l d. Setting b = z, d = u, 
and B = A~ l , we obtain 

(u'u) (length of projection) 2 = (z'u) 2 =£ (z'A~ l z)(u'Au) 

< c 2 u'Au for all z:z'A H z < c 2 

The choice z = cAu/ Vu'Au yields equalities and thus gives the maximum shadow, 
besides belonging to the boundary of the ellipsoid. That is, z'A _1 z = c^'Au/u'Au 
= c 2 for this z that provides the longest shadow. Consequently, the projection of the 
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ellipsoid onu is c Vu 7 ^ u/u'u, and its length is cVu'Au/u'u. With the unit vector 
e u u/v u u, the projection extends 


Vc e u Ae ll -Vu' Au units alone u 

Vu'u 

The projection of the ellipsoid also extends the same length in the direction -u. 


or 

.'A-'zse' implies that (U'a)'(U'AU)-‘(U'a) * c> forallU 

”™A-"™‘(t-P)A-^ e ;' wri " *■*-—(*-^)'(A-«.) and A-N 

z'A -1 z = (A’ 1/2 z)'(A~V 2 z ) 

= (PA-^z + (I - P)A- 1 /2 z) '( PA -i/2 z + (I _ p )A -l/2 z) 

= (PA- 1/2 z)'(PA-‘/ 2 z) + ((I - P)A-1/V((I - PjA-^z) 

- Z ' A ' 1/2P 'PA- 1/2 z = z'A-V2 PA -i/2 z = z'U(U'AU) _1 U'z (5A-1) 
Since z A : z < c 2 and U was arbitrary, the result follows. m 

confldence ellipse as a projection 



1 


Figure 5.13 The shadow of the 
ellipsoid z'A~ ! z < c 2 on the 
u i,u 2 plane is an ellipse. 
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Projection on a plane is simplest when the two vectors Ml and u 2 determining 
the plane are first converted to perpendicular vectors of unit leng . (See 

Result 2A.3.) 

Result 5A 3 Given the ellipsoid {rz'A^z £ c 2 } and two perpendicular unit 

Result 5A.3. U projection (or shadow) of {z’A^z £ c 2 } on the Ul ,u 2 

vectors u x and u 2 , the projection ^ j \ . < c 2\ where 

plane results in the two-dimensional ellipse {(U z) (U A ) ( h 

U = [ui i u 2 ]. 


Proof. By Result 2A.3, the projection of a vector z on the u,, u 2 plane is 


(„iz)u 1 + («2z)u 2 = [u 1 i« 2 ][^J = UU'z 


r h piijnsoid fz* i'A~ l z < c 2 } consists of all UU'z with 

V'gTf “™1l«.».»coJlh«e.U'.of.h„ proj.clion U(U’J. L=<*b* 

' fc.pH'.sa SO that UU'a belongs to the shadow of the elhpsotd. 

By Result 5A.2, 

(U'z)'(U'AUr l (U’z) s c 2 

so the ellipse {(U'z)' (U' AUf 1 (U'z) £ c 2 } contains the coefficient vectors for the 

>b Tt£a in the »t pl»= whosecoefficients a belong to,he e.hpse 

{a'(lT AUf’a sc 2 } Al we set z = AU(U'AU) a, it follows that 

U'z = U'AU(U'AU) _1 a = a 

and 

z'A-iz = a'(U’AU) _1 U'AA _1 AU(U'AU) -1 a = a'(U'AU) -, a ^ c 2 

Thm U’z belongs to the coefficient vector ellipse, and z belongs to the ellipsoid 
?A i; z < c 2 CoLequently, the ellipse contains only coefficient vectors from the 
projection of {z: z’A _1 z £ c 2 } onto the u„ u 2 plane. 

Remark Projecting the ellipsoid z'A^z s c 2 first to the u„ u 2 plane and then to 
^ T *’ ;c the same as projecting it directly to the line determined by Uj. In the 
context 3 confidence ellipsoids, the shadows of the two-dimensional ellipses give 
the single component intervals. 

Remark. Results 5A.2 and 5A.3 remain valid if U = [ui,....«,] consists of 
2 < q ^ p linearly independent columns. 
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E xercises _ 

5.1. (a) Evaluate T 2 , for testing H 0 : jtt' = [7, 11], using the data 

~2 12 " 

Y _ 8 9 
A ‘ 6 9 

_8 10 _ 

(b) Specify the distribution of T 2 for the situation in (a). 

(c) Using (a) and (b),test H 0 at the a = .05 level. What conclusion do you reach? 

5.2. Using the data in Example 5.1, verify that T 2 remains unchanged if each observation 
Xy, j = 1,2, 3; is replaced by Cx ; , where 



Note that the observations 



yield the data matrix 

"(6-9) (10-6) (8-3)1' 

J6 + 9) (10+6) (8 + 3)_ 

5.3. (a) Use expression (5-15) to evaluate T 2 for the data in Exercise 5.1. 

(b) Use the data in Exercise 5.1 to evaluate A in (5-13). Also, evaluate Wilks’ lambda. 

5.4. Use the sweat data in Table 5.1. (See Example 5.2.) 

(a) Determine the axes of the 90% confidence ellipsoid for / 1 . Determine the lengths of 
these axes. 

(b) Construct Q-Q plots for the observations on sweat rate, sodium content, and 
potassium content, respectively. Construct the three possible scatter plots for pairs 
of observations. Does the multivariate normal assumption seem justified in this 
case? Comment. 

5.5. The quantities x, S, and S" 1 are given in Example 5.3 for the transformed microwave- 
radiation data. Conduct a test of the null hypothesis H G : fi’ ~ [.55, .60] at the a - .05 
level of significance. Is your result consistent with the 95% confidence ellipse for /jl pic¬ 
tured in Figure 5.1? Explain. 

5.6. Verify the Bonferroni inequality in (5-28) for m = 3. 

Hint: A Venn diagram for the three events C x , C 2 , and C 3 may help. 

5.7. Use the sweat data in Table 5.1 (See Example 5.2.) Find simultaneous 95% T 2 confi¬ 
dence intervals for fi t , fju 2i and /i, 3 using Result 5.3. Construct the 95% Bonferroni inter¬ 
vals using (5-29). Compare the two sets of intervals. 
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5.8. 


P « 231 we know that T 2 is equal to the largest squared univariate /-value 

From (5-23), i inear combination a'x, with a = S (x - Mo)- Using the 

constructed from h jn E%eTC \<J 5 5( evaluate a for the transformed 

mTrola m ve E r X aXt,on data. Verify that the lvalue computed with this a is equal to T 1 

in Exercise 5.5. 

. , Q n ^ tlira i is t for the Alaska Fish and Game department, studies gri22ly 

5 ' 9 ' Swhh the goal of maintaining a healthy population. Measurements on n = 61 bears 
provided the following summary statistics (see also Exercise 8.23). 

Variable 


Sample 
mean 5 


Weight 

(kg) 

Body 

length 

(cm) 

Neck 

(cm) 

Girth 

(cm) 

Head 

length 

(cm) 

Head 

width 

(cm) 

95.52 

164.38 

55.69 

93.39 

17.98 

31.13 


Covariance matrix 


S = 


3266.46 

1343.97 

731.54 

1175.50 

162.68 

238.37 


1343.97 

721.91 

324.25 

53735 

80.17 

117.73 


731.54 

324.25 

179.28 

281.17 

39.15 

56.80 


1175.50 

537.35 

281.17 

474.98 

63.73 

94.85 


162.68 

80.17 

39.15 

63.73 

9.95 

13.88 


238.37 

117.73 

56.80 

94.85 

13.88 

21.26 


(a) Obtain the large sample 95% simultaneous confidence intervals for the six popula¬ 
tion mean body measurements. . 

(b) Obtain the large sample 95% simultaneous confidence ellipse for mean weight and 

, , obtain dte 95% Bonferroni confidence intervals for the six means in Part a. 

h Construct the 95% Bonferroni confidence rectangle for the mean 

(d) weigh/anc^ meangirth using m « 6. Compare this rectangle with the confidence 

ellipse in Part b. 

(e) Obtain the 95% Bonferroni confidence interval for 

mean head width - mean head length 

__ s , 1 - 7 to allow for this statement as well as statements about each 
using m - o -t i 

individual mean. 

5.0 Refer to the bear growth data in Example 1.10 (see Table 1.4). Restrict your attention to 
^ population means 

fhl ReferToPart a. Obtain the 95% T 2 simultaneous confidence intervals for the three 

successive yearly increases in mean length. ,h from 2 to 3 

(cl Obtain the 95% T 2 confidence ellipse for the mean increase in length from 
(C) Sand the mean increase in length from 4 to 5 years 
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(d) Refer to Parts a and b. Construct the 95% Bonferroni confidence intervals for the 
set consisting of four mean lengths and three successive yearly increases in mean 
length. 

(e) Refer to Parts c and d. Compare the 95% Bonferroni confidence rectangle for the 
mean increase in length from 2 to 3 years and the mean increase in length from 4 to 
5 years with the confidence ellipse produced by the T 2 -procedure. 

5.11. A physical anthropologist performed a mineral analysis of nine ancient Peruvian hairs. 

The results for the chromium (*j) and strontium (* 2 ) levels, in parts per million (ppm), 

were as follows: 


*i(Cr) 

.48 

40.53 

2.19 

.55 

.74 

.66 

.93 

.37 

.22 

* 2 (St) 

12.57 

73.68 

11.13 

20.03 

20.29 

.78 

4.64 

.43 

1.08 


Source: Benfer and others, “Mineral Analysis of Ancient Peruvian Hair ''American 
Journal of Physical Anthropology, 48 , no. 3 (1978), 277-282. 


It is known that low levels (less than or equal to .100 ppm) of chromium suggest the 

presence of diabetes, while strontium is an indication of animal protein intake. 

(a) Construct and plot a 90% joint confidence ellipse for the population mean vector 

fx' — assuming that these nine Peruvian hairs represent a random sample 

from individuals belonging to a particular ancient Peruvian culture. 

(b) Obtain the individual simultaneous 90% confidence intervals for /ij and by “pro¬ 
jecting” the ellipse constructed in Part a on each coordinate axis. (Alternatively, we 
could use Result 5.3.) Does it appear as if this Peruvian culture has a mean strontium 
level of 10? That is, are any of the points arbitrary, 10) in the confidence regions? 
Is [.30,10]' a plausible value for /t? Discuss. 

(c) Do these data appear to be bivariate normal? Discuss their status with reference to 
Q-Q plots and a scatter diagram. If the data are not bivariate normal, what implica¬ 
tions does this have for the results in Parts a and b? 

(d) Repeat the analysis with the obvious “outlying” observation removed. Do the infer¬ 
ences change? Comment. 

5.12. Given the data 




3 

4 


5 


6 0 
4 3 

8 3 


with missing components, use the prediction-estimation algorithm of Section 5.7 to 
estimate /t and X. Determine the initial estimates, and iterate to find the first revised 
estimates. 

5.13. Determine the approximate distribution of -nln(| X |/|X 0 |) for the sweat data in 
Thble 5.1. (See Result 5.2.) 

5.14* Create a table similar to Table 5.4 using the entries (length of one-at-a-time ^interval)/ 
(length of Bonferroni f-interval). 
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Exercises 5.15 , 5.16, and 5.17 refer to the following information: 

Frequently, some or all of the population characteristics of interest are in the form of 
attributes. Each individual in the population may then be described in terms of the 
attributes it possesses. For convenience, attributes are usually numerically coded with re¬ 
spect to their presence or absence. If we let the variable X pertain to a specific attribute, 
then we can distinguish between the presence or absence of this attribute by defining 

_ f 1 if attribute present 
\0 if attribute absent 

In this way, we can assign numerical values to qualitative characteristics. 

When attributes are numerically coded as 0-1 variables, a random sample from the 
population of interest results in statistics that consist of the counts of the number of 
sample items that have each distinct set of characteristics. If the sample counts are 
large, methods for producing simultaneous confidence statements can be easily adapted 
to situations involving proportions. 

We consider the situation where an individual with a particular combination of 
attributes can be classified into one of q + 1 mutually exclusive and exhaustive 
categories. The corresponding probabilities are denoted by p 1 , p 2 , • * •, p q , Pq+ 1 - Since 
the categories include all possibilities, we take p q+[ = 1 - (/?] + Pi + ‘ + p q ). An 

individual from category k will be assigned the ((q + 1) X 1) vector value [0,..., 0, 
1 , 0 ,..., 0]'with 1 in the fcth position. 

The probability distribution for an observation from the population of individuals in 
q + 1 mutually exclusive and exhaustive categories is known as the multinomial distrib¬ 
ution. It has the following structure: 


Category 


Outcome (value) 


Probability 

(proportion) 



Let Xy,/ = 1,2,..., n, be a random sample of size n from the multinomial 
distribution. 

The ki h component, X )k , of X ; is 1 if the observation (individual) is from category k 
and is 0 otherwise. The random sample X Y , X 2 ,... ,X„ can be converted to a sample 
proportion vector, which, given the nature of the preceding observations, is a sample 
mean vector. Thus, 





n 
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and 

°Tl ^12 

Cov(p) = -Cov(X,) = -X = - °’. 21 °’? 2 

_°T,g + l cr 2,q+l 

For large n , the approximate sampling distribution of p is provided by the central limit 
theorem. We have 

Vn (p - p) is approximately jV(0, X) 

where the elements of X are cr kk — p k ( 1 — p k ) and a ik ~ ~PiP k - The normal approx¬ 
imation remains valid when <r kk is estimated by & kk — p k ( 1 - p k ) and a ik is estimated 
by & lk = ~PiP k> i * k. 

Since each individual must belong to exactly one category, X q+1 j ~ 
1 - (X tj + X 2 / + •••+ x qj ), so Pq+i = 1 (pi + P 2 + ■' ■ + Pq), and as a result, X 
has rank q. The usual inverse of X does not exist, but it is still possible to develop simul¬ 
taneous 100(1 — a)% confidence intervals for all linear combinations a'p. 

Result. Let X 1? X 2 , • •• ,X n be a random sample from a q + 1 category multinomial 
distribution with P[X jk — 1] = p k , k ~ 1,2,... ,q + 1, j = 1,2Approximate 
simultaneous 100(1 - a)% confidence regions for all linear combinations a'p 
= aip { + a 2 p 2 + • * * + a q+x p q+l are given by the observed values of 



provided that n — q is large. Here p = (1 /n) 2 Xy, and X = {o'ik} is a (q + 1) X (q + 1) 

/- 1 ^ ^ 

matrix with 6- kk — p k ( 1 — p k ) and a lk = —p t p k , i * k. Also, Xq{ a ) is upper 
(100a) th percentile of the chi-square distribution with q d.f. ■ 

In this result, the requirement that n - q is large is interpreted to mean np k is 
about 20 or more for each category. 

We have only touched on the possibilities for the analysis of categorical data. Com¬ 
plete discussions of categorical data analysis are available in [1] and [4], 

5.15. Let Xji and X )k be the /th and klh components, respectively, of X } . 

(a) Show that/ l,- = £’(A r ;/ ) = p, anda,, = Var (Xjj) ~ p,( 1 - pf),i = 1,2,... ,p. 

(b) Show that a ik = Cov(Jf ;i , Xj k ) = ~PiP ki i * k. Why must this covariance neces¬ 
sarily be negative? 

5.16. As part of a larger marketing research project, a consultant for the Bank of Shorewood 
wants to know the proportion of savers that uses the bank’s facilities as their primary ve¬ 
hicle for saving. The consultant would also like to know the proportions of savers who 
use the three major competitors: Bank B, Bank C, and Bank D. Each individual contact¬ 
ed in a survey responded to the following question: 
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Which bank is your primary savings bank? 

Bank of I D D I 


Another No 


Response: sho ^ewood | Bank B | Bank C | Ba ° k D | Bank | Savings 

A sample of n = 355 people with savings accounts produced the following counts 
when asked to indicate their primary savings banks (the people with no savings will bg 
ignored in the comparison of savers, so there are five categories): 



^^^(catego ry; 

"Observed 

number 

population 

^roportto n 

Observed sample 
proportion 


Bank of Shorewood 

Bank B 

Bank C 

Bank D 

A noth er b a nk T'lfljM 

105 

119 

56 

25 

50 *-“®| 

P\ 

P2 

Ph 

Pa 

(Pi + Pi + £3 + Pa) 

P ' 355 

Pi = -33 

P3 = 16 

Pa = -07 

1 


Let the population proportions be 

p v = proportion of savers at Bank of Shorewood 

p 2 ~ proportion of savers at Bank B 
p 3 = proportion of savers at Bank C 

p 4 = proportion of savers at Bank D 

1 “ [P\ + Pi + P3 + Pa) = proportion of savers at other banks 

(a) Construct simultaneous 95% confidence intervals for p x , p 2 , • ■ ■» P$ * 

(b) Construct a simultaneous 95% confidence interval that allows a comparison o t e 
Bank of Shorewood with its major competitor, Bank B. Interpret this t 

5 17 . In order to assess the prevalence of a drug problem among high school stud , 

particular city, a random sample of 200 students from the city s ive ses are 
were surveyed. One of the survey questions and the corresponding r p 
as follows: 

What is your typical weekly marijuana usage? 



I 


Construct 95% simultaneous confidence intervals for the three proportions p x , p 2 > and 

P3 = 1 ~ (Pi + P 2 )' 

The following exercises may require a computer 

5.18. Use the college test data in Table 5.2. (See Example 5.5.) 

(a) Test the null hypothesis // 0 : /*' = [500,50, 30] versus H y : ft' ^ [500, 50, 30] at the 
a = .05 level of significance. Suppose [500,50,30]' represent average scores for 
thousands of college students over the last 10 years. Is there reason to believe that the 
group of students represented by the scores in Table 5.2 is scoring differently? 
Explain. 

(b) Determine the lengths and directions for the axes of the 95% confidence ellipsoid for ft. 

(c) Construct Q-Q plots from the marginal distributions of social science and history, 
verbal, and science scores. Also, construct the three possible scatter diagrams from 
the pairs of observations on different variables. Do these data appear to be normally 
distributed? Discuss. 

5.19. Measurements of x x = stiffness and x 2 = bending strength for a sample of n = 30 pieces 

of a particular grade of lumber are given in Thble 5.11. The units are pounds/(inches) 2 . 

Using the data in the table, 




Category _ 


None 

Moderate 
(1-3 joints) 

Heavy 

(4 or more joints) 

Number of 
responses 

117 

62 

2! 



(a) Construct and sketch a 95% confidence ellipse for the pair [^1,^2] ’ where 
Pi ~ £^(^1) and p 2 ~~ E(X 2 ). 

(b) Suppose p l0 = 2000 and p 2 o ~ 10,000 represent “typical” values for stiffness and 
bending strength, respectively. Given the result in (a), are the data in Table 5.11 con¬ 
sistent with these values? Explain. 
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ence to Q Q P | th (in millimeters) and x 2 = wmg lengthen ^ 

5.20: A wildlife ecologist measured x, {emalebook . bl n e d kites.These data are displayed in J 
millimeters) for a sample of n 
Table 5.12. Using the data in th 



, -o piiincp for the population means /r i and /C; . 

(„ to, 

S S5Xe tt ttoS^ f ° r thC mean ^ ' ength aDd mean W ' nS 

the female birds? Explain. ^ fof and ^ and the 95% Bonferroni 

(b) Construct the simultaneous 95/o ^ ^ ofl £ ervals what advantage, if any, do 

intervals for *i, an I Mz-P Bonferron ; intervals? . 

theTMntervalshaveov r a ^ ^ion model? Explain with 

(c) Is the "."Lascatterdiagmm. 

reference Q Q in Table 18 construct the 95% Bonferroni 

find the 95% simultaneous intervals. 

Compare the two sets of intervals. 


5.22 


Compare the two sets chapter 6 is reproduced in Table 5.13. 

;. A portion of the data contented^ n ™e6lCwith transporting milk from farms to dairy 
These data represent vanous ws ^ ^ multivatiate obserV ations for gasoline tructe 
plants for gasoline trucks On y identified as outliers from the full data se 

are given. Observations 9 and 21 have 
36 observations. (See [2].) 


Table 5.13 Milk Transportation-Cost Data 

Fuel (x x ) 

Repair (x 2 ) 

Capital (x 3 ) 

16.44 

12.43 

11.23 

7.19 

2.70 

3.92 

9.92 

1.35 

9.75 

4.24 

5.78 

7.78 

11.20 

5.05 

10.67 

14.25 

5.78 

9.88 

13.50 

10.98 

10.60 

13.32 

14.27 

9.45 

29.11 

15.09 

3.28 

12.68 

7.61 

10.23 

7.51 

5.80 

8.13 

9.90 

3.63 

9.13 

10.25 

5.07 

10.17 

11.11 

6.15 

7.61 

12.17 

14.26 

14.39 

10.24 

2.59 

6.09 

10.18 

6.05 

12.14 

8.88 

2.70 

12.23 

12.34 

7.73 

11.68 

8.51 

14.02 

12.01 

26.16 

17.44 

16.89 

12.95 

8.24 

7.18 

16.93 

13.37 

17.59 ' 

14.70 

10.7# 

14.58 

10.32 

5.16 

17.00 


(a) Construct Q-Q plots of the marginal distributions of fuel, repair, and capital costs. 
Also, construct the three possible scatter diagrams from the pairs of observations on 
different variables. Are the outliers evident? Repeat the Q-Q plots and the scatter 
diagrams with the apparent outliers removed. Do the data now appear to be nor¬ 
mally distributed? Discuss. 

(b) Construct 95% Bonferroni intervals for the individual cost means. Also, find the 
95% ^-intervals. Compare the two sets of intervals. 

5.23. Consider the 30 observations on male Egyptian skulls for the first time period given in 
Table 6.13 on page 349. 

(a) Construct Q-Q plots of the marginal distributions of the maxbreath, basheight, 
baslength and nasheight variables. Also, construct a chi-square plot of the 
multivariate observations. Do these data appear to be normally distributed? 
Explain. 

(b) Construct 95% Bonferroni intervals for the individual skull dimension variables. 
Also, find the 95% T 2 -intervals. Compare the tw , o sets of intervals. 

5.24. Using the Madison, Wisconsin, Police Department data in Table 5.8, construct individual 
X charts for x 3 = holdover hours and x 4 — COA hours. Do these individual process 
characteristics seem to be in control? (That is, are they stable?) Comment. 
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• c ox rt ■ „ .h„ on the holdover and COA overtime hours,con- 

5.25. Refer to Exercise 5.24. Using ‘ Does (he process represented by the bivariate ; 

struct a quality ellipse and a • ,_ t j js jt sta ble?) Comment. Do you learal 

something fro^he'multivariaie couttol M M ~ « HI-™"«* '■«*< 

X -charts? 

5.26. Construct a Dehart using “S 

= extraordinary even t 5.8 of Example 5.10. Does plotting 

Table 5.8. Compare this char wi hanoe vour conclusion about process stability?^ 

j 2 wi ,h an additional characteristic change your conclusion an 

Explain. ' 1 

5.27. Using the data on *3 = holdover houreand^ = 

:,l^3/Kc U T.,L <Z . stable P»» Interpret the result. 

5.28 AS P... Of . stud, Of it. sheet me,.I 

uses sensors that record the delation r complete and the last- 

given in Tab e cases Use these cases to estimate S and x. 

(3) SSSSS- 1 !ctrtusiS of thevariab.es. Inc.ude all 50 cases. 

(b) Which individual locations seem to show a cause for concern. 

. , • p vf . rri c e 5 28 These ate all measured as deviations from 

5.29 Refer to the car body data in Ex • hypothesis that the mean vector is zero, 

target value so it is appropriate to test the nun nyp 

Using the first 30 cases, test H a \ n - 0 at a 

yyo ,* „„ m 

<*> ““ re’h :Sh ^fo“re£-a. *>».- *« ***•»■***■ 

urn minus natural gas. , f in , erva ls for the mean consumption 

<b> s"is StBESSJ:- °< »* <«• »*»««»““■ m 

natural gas. Compare with your results for Part a. 

(b) Onthesame scale.ftnd the 95% Bonferroni confidence intervals for the twocompo- 
nent means. 
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TABLE 5.14 Car Body Assembly Data 


Index 

X\ 

*2 

*3 

x 4 

*5 

X 6 


-0.12 

0.36 

0.40 

0.25 

1.37 

-0.13 

2 

-0.60 

-0.35 

0.04 

-0.28 

-0.25 

-0.15 

3 

-0.13 

0.05 

0.84 

0.61 

1.45 

0.25 

4 

-0.46 

-0.37 

0.30 

0.00 

-0.12 

-0.25 

5 

-0.46 

-0.24 

0.37 

0.13 

0.78 

-0.15 

6 

-0.46 

-0.16 

0.07 

0.10 

1.15 

-0.18 

7 

-0.46 

-0.24 

0.13 

0.02 

0.26 

-0.20 

8 

-0.13 

0.05 

-0.01 

0.09 

-0.15 

-0.18 

9 

-0.31 

-0.16 

-0.20 

0.23 

0.65 

0.15 

10 

-0.37 

-0.24 

0.37 

0.21 

1.15 

0.05 

11 

-1.08 

-0.83 

-0.81 

0.05 

0.21 

0.00 

12 

-0.42 

-0.30 

0.37 

-0.58 

0.00 

-0.45 

13 

-0.31 

0.10 

-0.24 

0.24 

0.65 

0.35 

14 

-0.14 

0.06 

0.18 

-0.50 

1.25 

0.05 

15 

-0.61 

-0.35 

-0.24 

0.75 

0.15 

-0.20 

16 

-0.61 

-0.30 

-0.20 

-0.21 

-0.50 

-0.25 

17 

-0.84 

-0.35 

-0.14 

-0.22 

1.65 

-0.05 

18 

-0.96 

-0.85 

0.19 

-0.18 

1.00 

-0.08 

19 

-0.90 

-0.34 

-0.78 

-0.15 

0.25 

0.25 

20 

-0.46 

0.36 

0.24 

-0.58 

0.15 

0.25 

21 

-0.90 

-0.59 

0.13 

0.13 

0.60 

-0.08 

22 

-0.61 

-0.50 

-0.34 

-0.58 

0.95 

-0.08 

23 

-0.61 

-0.20 

-0.58 

-0.20 

1.10 

0.00 

24 

-0.46 

-0.30 

-0.10 

-0.10 

0.75 

-0.10 

25 

-0.60 

-0.35 

-0.45 

0.37 

1.18 

-0.30 

26 

-0.60 

-0.36 

-0.34 

-0.11 

1.68 

-0.32 

27 

-0.31 

0.35 

-0.45 

-0.10 

1.00 

-0.25 

28 

-0.60 

-0.25 

-0.42 

0.28 

0.75 

0.10 

29 

-0.31 

0.25 

-0.34 

-0.24 

0.65 

0.10 

30 

-0.36 

-0.16 

0.15 

-0.38 

1.18 

-0.10 

31 

-0.40 

-0.12 

-0.48 

-0.34 

0.30 

-0.20 

32 

-0.60 

-0.40 

-0.20 

0.32 

0.50 

0.10 

33 

-0.47 

-0.16 

-0.34 

-0.31 

0.85 

0.60 

34 

-0.46 

-0.18 

0.16 

0.01 

0.60 

0.35 

35 

-0.44 

-0.12 

-0.20 

-0.48 

1.40 

0.10 

36 

-0.90 

-0.40 

0.75 

-0.31 

0.60 

-0.10 

37 

-0.50 

-0.35 

0.84 

-0.52 

0.35 

-0.75 

38 

-0.38 

0.08 

0.55 

-0.15 

0.80 

-0.10 

39 

-0.60 

-0.35 

-0.35 

-0.34 

0.60 

0.85 

40 

0.11 

0.24 

0.15 

0.40 

0.00 

-0.10 

41 

0.05 

0.12 

0.85 

0.55 

1.65 

-0.10 

42 

-0.85 

-0.65 

0.50 

0.35 

0.80 

-0.21 

43 

-0.37 

-0.10 

-0.10 

-0.58 

1.85 

-0.11 

44 

-0.11 

0.24 

0.75 

-0.10 

0.65 

-0.10 

45 

-0.60 

-0.24 

0.13 

0.84 

0.85 

0.15 

46 

-0.84 

-0.59 

0.05 

0.61 

1.00 

0.20 

47 

-0.46 

-0.16 

0.37 

-0.15 

0.68 

0.25 

48 

-0.56 

-0.35 

-0.10 

0.75 

0.45 

0.20 

49 

-0.56 

-0.16 

0.37 

-0.25 

1.05 

0.15 

50 

-0.25 

-0.12 

-0.05 

-0.20 

1.21 

0.10 


Source: Data Courtesy of Darek Ceglarek. 
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Chapter 



Comparisons of Several 
Multivariate Means 


6.1 Introduction 

The ideas developed in Chapter 5 can be extended to handle problems involving the 
comparison of several mean vectors. The theory is a little more complicated and 
rests on an assumption of multivariate normal distributions or large sample sizes. 
Similarly, the notation becomes a bit cumbersome. To circumvent these problems, 
we shall often review univariate procedures for comparing several means and then 
generalize to the corresponding multivariate cases by analogy. The numerical exam¬ 
ples we present will help cement the concepts. 

Because comparisons of means frequently (and should) emanate from designed 
experiments, we take the opportunity to discuss some of the tenets of good experi¬ 
mental practice. A repeated measures design, useful in behavioral studies, is explicitly 
considered, along with modifications required to analyze growth curves. 

We begin by considering pairs of mean vectors. In later sections, we discuss sev¬ 
eral comparisons among mean vectors arranged according to treatment levels. The 
corresponding test statistics depend upon a partitioning of the total variation into 
pieces of variation attributable to the treatment sources and error. This partitioning 
is known as the multivariate analysis of variance (MANOVA). 

6.2 Paired Comparisons and a Repeated Measures Design 

Paired Comparisons 

Measurements are often recorded under different sets of experimental conditions 
to see whether the responses differ significantly over these sets. For example, the 
efficacy of a new drug or of a saturation advertising campaign may be determined by 
comparing measurements before the “treatment” (drug or advertising) with those 
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after the treatment. In other situations, two or more treatments can be administered 
to the same or similar experimental units, and responses can be compared to asse$| 
the effects of the treatments. 

One rational approach to comparing two treatments, or the presence and 
sence of a single treatment, is to assign both treatments to the same or identical uni 
(individuals, stores, plots of land, and so forth). The paired responses may then b|| 
analyzed by computing their differences, thereby eliminating much of the influent 
of extraneous unit-to-unit variation. ]_ 

In the single response (univariate) case, let X jx denote the response t| 
treatment 1 (or the response before treatment), and let X j2 denote the response g 
treatment 2 (or the response after treatment) for the yth trial. That is, (X jU Xj 2 1 
are measurements recorded on the /th unit or /th pair of like units. By design, the^ 
n differences , ‘ 

Dj = Xji - Xj 2> / = 1,2,..., n (6-1), 

should reflect only the differential effects of the treatments. 

Given that the differences D } in (6-1) represent independent observations from- 
an N(S , a 2 d ) distribution, the variable 


D = -'£ J D j and s 2 d = ^ ^ (Dj - D ) 2 (6-3) 

n P\ n-\ r \ 

has a /-distribution with n - 1 d.f. Consequently, an a-level test of 
H 0 :6 = 0 (zero mean difference for treatments) 

versus 

8 * 0 

may be conducted by comparing |/| with t n -[(a/2) the upper 100(a/2)th per¬ 
centile of a /-distribution with n - 1 d.f. A 100(1 - a)% confidence interval for the 
mean difference 8 = E{X n ~ X }2 ) is provided the statement 

d - t n -,{a/2)^= £ S £ d + t„- 1(«/2)^ ( 64 ) 

(For example, see [11].) . . 

Additional notation is required for the multivariate extension of the paired- 
comparison procedure. It is necessary to distinguish between p responses, two treat¬ 
ments, and n experimental units. We label the p responses within the jth unit as 

X 1 j , = variable 1 under treatment 1 
*1,2 = variable 2 under treatment 1 


X\j p = variable p under treatment 1 
X 2 j\ = variable 1 under treatment 2 

X 2j2 = variable 2 under treatment 2 
X 2 jn - variable p under treatment 2 
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and the p paired-difference random variables become 

D n = *1/1 - *2/1 

Dj 2 - X\j 2 - X 2j2 

Djp — X\ j P — X 2 j p 

Let D) = f Dj U D j2i ..., Dj p ], and assume, for / = 1,2that 

*r 

S 2 

and Cov(D y ) = 


E(Dj) = 8 = 


(6-5) 


(6-6) 


If, in addition, D!, D 2 ,..., D„ are independent N p (8 y 2 d ) random vectors, infer¬ 
ences about the vector of mean differences 5 can be based upon a T 2 -statistic. 
Specifically, 

T 2 = n(D - 5)V(D - 8) (6-7) 

where 

D = ^ 2 D, and S, = -L- ± (D y - D)(D y - D)' (6-8) 

7 = 1 n 1 /=1 

Result 6.1. Let the differences D 1 ,D 2 ,-..,D rt be a random sample from an 
N p (8 , S d ) population.Then 

r 2 = n(D - 5)V(D - 8) 

is distributed as an [ (n - l)p/(n - p)]F p n ^ p random variable,.whatever the true 8 
and Xd- 

If n and n - p are both large, T 2 is approximately distributed as a Xp random 
variable, regardless of the form of the underlying population of differences. 

Proof. The exact distribution of T 2 is a restatement of the summary in (5-6), with 
vectors of differences for the observation vectors The approximate distribution of 
T 2 , for n and n - p large, follows from (4-28). ■ 

The condition 8 = 0 is equivalent to “no average difference between the two 
treatments.” For the /th variable, 6/ > 0 implies that treatment 1 is larger, on aver¬ 
age, than treatment 2. In general, inferences about 8 can be made using Result 6.1. 


Given the observed differences d' = [d jX , d j2 , ..., dj p ], j = 1 , 2,. .., n, corre¬ 
sponding to the random variables in (6-5), an a-level test of H 0 : 8 - 0 versus 
H x : 8 * 0 for an N p (8 y X d ) population rejects H 0 if the observed 

T 2 = nd'Sj'd > ~ ^ />,„-„(«) 

where F p n _ p (a) is the upper (100a)th percentile of an F-distribution with p 
and n - p d.f. Here d and S d are given by (6-8). 
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A 100(1 - a)% confidence region for 8 consists of all 6 such that 

(d - «)V(* - «) s ( n n (n Fp,n-A a ) < 6 ' 9 ) 

Also, 100(1 - a)% simultaneous confidence intervals for the individual mean 
differences Si are given by 

5*: 5, ± 

where d, is the ith element of d and s\ is the ith diagonal element of S rf . 

For r, - p large, [(n - l)p/(« - and normallt y 

need not be assumed. . 

The Bonferroni 100(1 - o)% simufomeous confidence intervals for the 



individual mean differences are 


d l ± t n -\ 


WVn 


(6-10a) 


where ? n _,(«/2p) is the upper 100(«/2p)th percentile of a ^distribution with 


n - 1 d.f. 


Example 6.1 (Cheeking for a mean difference with paired observations) Municipal 
wastewater treatment plants are required by law to monitor their discharges into 
rivers and streams on a regular basis. Concern about the reliability of data from one 
of these self-monitoring programs led to a study in which samples of effluent were 
divided and sent to two laboratories for testing. One-half of each sample was sent to 
the Wisconsin State Laboratory of Hygiene, and one-half was sent to a private com¬ 
mercial laboratory routinely used in the monitoring program. Measurements of bio¬ 
chemical oxygen demand (BOD) and suspended solids (SS) were obtained for 
n = 11 sample splits,from the two laboratories.The data are displayed in Table 6.1. 
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Do the two laboratories’ chemical analyses agree? If differences exist, what is 
their nature? 

The r 2 -statistic for testing Ho: 8' = [5j, 5 2 ] = [0,0] is constructed from the 
differences of paired observations: 


d}\ - *i)i 

- *2)1 

-19 

-22 

-18 

-27 -4 -10 

-14 

17 

9 

4 

-19 

dji ~ *1)2 

~ *2)2 

12 

10 

42 

15 -1 11 

-4 

60 

-2 

10 

-7 



Taking a = .05, we find that [p(n ~ 1 )/{n - p)}F p ^ p (: 05) = [2(10)/9]/ 7 2 , 9 (.05) 
= 9.47. Since T 2 = 13.6 > 9.47, we reject H 0 and conclude that there is a nonzero 
mean difference between the measurements of the two laboratories. It appears, 
from inspection of the data, that the commercial lab tends to produce lower BOD 
measurements and higher SS measurements than the State Lab of Hygiene. The 
95% simultaneous confidence intervals for the mean differences 6! and S 2 can be 
computed using (6-10). These intervals are 



The 95% simultaneous confidence intervals include zero, yet the hypothesis Ho: 6=0 
was rejected at the 5% level. What are we to conclude? 

The evidejjce points toward real differences. The point 6 = 0 falls outside 
the 95% confidence region for 6 (see Exercise 6.1), and this result is consistent 
with the Latest. The 95% simultaneous confidence coefficient applies to the 
entire set of intervals that could be constructed for all possible linear com¬ 
binations of the form + a 2 S 2 . The particular intervals corresponding to the 
choices {a\ = 1, a 2 = 0) and ( a\ = 0, a 2 = 1) contain zero. Other choices of a\ 
and a 2 will produce simultaneous intervals that do not contain zero. (If the 
hypothesis H 0 : 6 = 0 were not rejected, then all simultaneous intervals would 
include zero.) 

The Bonferroni simultaneous intervals also cover zero. (See Exercise 6.2.) 
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Our analysis assumed a normal distribution for the D,. In fact, the situation is 
further complicated by the presence of one or, possibly, two outliers. (See Exercise? 
6.3.) These data can be transformed to data more nearly normal, but with such a 
small sample, it is difficult to remove the effects of the outlier(s). (See Exercise 6.4.)? 

The numerical results of this example illustrate an unusual circumstance that 
can occur when,making inferences. ^ 

The experimenter in Example 6.1 actually divided a sample by first shaking it and 
then pouring it rapidly back and forth into two bottles for chemical analysis. This was 
prudent because a simple division of the sample into two pieces obtained by pouring 
the top half into one bottle and the remainder into another bottle might result in more 
suspended solids in the lower half due to setting.The two laboratories would then not 
be working with the same, or even like, experimental units, and the conclusions would 
not pertain to laboratory competence, measuring techniques, and so forth. 

Whenever an investigator can control the assignment of treatments to experi¬ 
mental units, an appropriate pairing of units and a randomized assignment of treats 
ments can enhance the statistical analysis. Differences, if any, between supposedly 
identical units must be identified and most-alike units paired. Further, a random as¬ 
signment of treatment 1 to one unit and treatment 2 to the other unit will help elim¬ 
inate the systematic effects of uncontrolled sources of variation. Randomization can 
be implemented by flipping a coin to determine whether the first unit in a pair re¬ 
ceives treatment 1 (heads) or treatment 2 (tails). The remaining treatment is then 
assigned to the other unit. A separate independent randomization is conducted for 
each pair. One can conceive of the process as follows: 

Experimental Design for Paired Comparisons 


Like pairs of 
experimental 


Treatments 
] and 2 
assigned 
at random 


Treatments 
I and 2 
assigned 
at random 


Treatments 
I and 2 
assigned 
at random 


Treatments 
► 1 and 2 

assigned 
at random 


We conclude our discussion of paired comparisons by noting that d and S^, and 
hence T 2 , may be calculated from the full-sample quantities x and S. Here x is the 
2p X 1 vector of sample averages for the p variables on the two treatments given by 

*' = [xn,x l2 ,...,X]p,x 2 i,X22,---'X2p] ( 6 - 11 ) 

and S is the 2 p x 2 p matrix of sample variances and covariances arranged as 

S 11 Sl2 

c _ ( p x p) ( p x p ) ( 6 - 12 ) 

S 21 S 22 

L(pxp) (p*p)_ 
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treatment^ ^ Sample Variances and “^nances for the p variables on 

orTJ o varth f COnta,ns « he Sam P Ie varianc « and covariances computed 

Svaria/ces co^n, m ! nt 2 FmaUy ’ S “ = S *> are the "“trices of sample 

variable" observat.ons on pairs of treatment 1 and treatment 2 

Defining the matrix 



”l 

0 ■ 

* 0 

-1 

0 .. 

♦ o" 

c = 

0 

1 • 

• 0 

0 

-1 •. 

• 0 

(p*2p) 

_0 

0 .. 

• 1 

0 

T 

0 .. 

* 


(p + l)st column 


we can verify (see Exercise 6.9) that 


d i = Cx ; , j = 1,2,» 
d = Cx and S d = CSC' 


T2 = rt * , C'(CSC') -1 Cx (6-15) 

and it is not necessary first to calculate the differences d t , d 2) . d On the other 
sumption d ‘ fferenCeS *“ * Check n0nDality and the as ‘ 

Each contrast is perpendicular to the vector V = [ 1 , 1 ,... lE = 0 t 

SS;S^ overa11 “ - * - - 


A Repeated Measures Design for Comparing Treatments 

Another generalization of the univariate paired /-statistic arises in situations where 

or «Sl S nLTunTr Pared Wi ' h *° “ ^ reSp ° nSe Variable - Each object 

^eTobse^ation is^ treatmen ‘ ° DCe ° Ver P™ds of time. 


J “ 1,2 
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For comparative purposes, we consider contrasts of the components of 
fi = E(Xj). These could be 


Ml - M3 


Ml “ Fq 


M 2 - Mi 
M3 “ M2 


1 0 0 

-1 1 0 

0 -1 1 


= C,M 


0 0 

0 0 al 2 


Both Ci and C 2 are called contrast matrices , because their q — 1 rows are linearly 
independent and each is a contrast vector. The nature of the design eliminates much 
of the influence of unit-to-unit variation on treatment comparisons. Of course, the 
experimenter should randomize the order in which the treatments are presented to 
each subject. 

When the treatment means are equal, C\ji = C 2 /t = 0. In general, the hypoth¬ 
esis that there are no differences in treatments (equal treatment means) becomes 
C n ~ 0 for any choice of the contrast matrix C. 

Consequently, based on the contrasts Cx 7 in the observations, we have means 
Cx and covariance matrix CSC', and we test C n = 0 using the F 2 -statistic 

T 2 = /i(Cx)'(CSC , r 1 Cx 


Test for Equality of Treatments in a Repeated Measures Design 

Consider an N q (n, X) population,and let C be a contrast matrix.An a-level test 
of H 0 : Cjjl — 0 (equal treatment means) versus H\i Cfi ^ 0 is as follows: 

Reject H 0 if 

T 2 = n(Cx)'(CSC') -1 Cx > (6-16) 

(n - q + 1) 

where F q -} >n „ q+l {a) is the upper (100a)th percentile of an F-distribution with 
q - 1 and n — q + 1 d.f. Here x and S are the sample mean vector and covari¬ 
ance matrix defined, respectively, by 

x = i|x y and S = ~^ T i(x,-x)(x y -x)' 

It can be shown that T 2 does not depend on the particular choice of C. 1 

] Any pair of contrast matrices Cj and C 2 must be related by C| = BC 2 , with B nonsingular. 
This follows because each C has the largest possible number, q ~ 1, of linearly independent rows, 
all perpendicular to the vector 1. Then (BC 2 )'(BC 2 SC 2 B')~ 1 (BC 2 ) = C 2 B'(B') -, (C 2 SC 2 ) _, B" J BC 2 
C 2 (C 2 SC 2 ) -1 C 2 , so T 2 computed with C 2 or Cj = BC 2 gives the same result. 
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is d,.t C l fi i!! Ce ,? 0n f ru°" traStS Cft ’ with M the mean of a normal population, 
is determined by the set of all C/j. such that 

n(Cx - C/t)'(CSC') -1 (Cx - C n) s — ~ ~ ri F , . 

(n-q + i) F ‘i-hn-q+l(a) (6-17) 

ctnfiHent nd f "7! de ®“ d in ^ Consequently, simultaneous 100(1 - «)% 

vent 7T Va S l°I ^ C ° ntraStS C> for an >- cont «st vectors of interest are 
given by (see Result 5A.1) 


c>: c x ± 


(n - l)(q - 1) 

(n - q + 1) 


^7-l.n-7+l(<*) 


( 7" g f 7 eq , Ual treatments in a repeated measures design) Improved 

studv H, 3re developed by first studying their effects on animals. In one 

minkJ 9 J gS h Wer 7 ln '77 glVen ‘ he drug P ent °6arbitol. Each dog was then ad- 
was aHHed C ° 2 3 ‘ each ° f two P ressuie levels. Next, halothane (H) 

between ia i ? admimstration of C °2 was repeated. Tire response, milliseconds 
between heartbeats, was measured for the four treatment combinations: 

Present 

Halothane 

Absent 

Low High 

CC >2 pressure 



Table 6.2 contains the four measurements for each of the 19 dogs, where 
Treatment 1 = high C0 2 pressure without H 
Treatment 2 = lowC0 2 pressure without H 
Treatment 3 = high C0 2 pressure with H 
Treatment 4 = low C0 2 pressure with H 

this analyze anesthetizing effects of C0 2 pressure and halothane from 

uus repeated-measures design. 

There are three treatment contrasts that might be of interest in the experiment. 
4, respectively'Then^ 4 COrreSp ° nd t0 the mean res Ponses for treatments 1,2,3, and 

/Halothane contrast representing the \ 

(M3 + M ~ (Mi + M 2 ) = difference between the presence and 
\ absence of halothane / 


(Ml + M 3 ) " (M 2 + M 4 ) 


(Mi + M 4 ) (m 2 + M 3 ) — 


C0 2 contrast representing the difference ^ 
between high and low C0 2 pressure ^ 

Contrast representing the influence 
of halothane on C0 2 pressure differences 
(H-C0 2 pressure “interaction”) 
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Table 6.2 Sleeping-Dog Data 


Treatment 


Source: Data courtesy of Dr. J.Atlee. 



With /i' = [Mi • M 21 ^ 4 ]. the contrast matrix C is 


368.21 

404.63 

479.26 

502.89 


It can be verified that 

' 209.31 
Cx = -60.05 

-12.79 


The data (see Table 6.2) give 


and S = 


- 1-1 1 1 
1-1 1-1 
1 - 1-1 1 


2819.29 

3568.42 7963.14 

2943.49 5303.98 6851.32 

2295.35 4065.44 4499.63 4878.99 


'9432.32 1098.92 927.62 

CSC' = 1098.92 5195.84 914.54 

927.62 914.54 7557.44 


and 


T 2 = n(Cx)'(CSC') ‘(Cx) = 19(6.11) = 116 



The first confidence interval implies that there is a halothane effect. The pres¬ 
ence of halothane produces longer times between heartbeats. This occurs at both 
levels of C0 2 pressure, since the H-C0 2 pressure interaction contrast, 
(mj + M 4 ) “ (m 2 ~ M 3 ), is not significantly different from zero. (See the third 
confidence interval.) The second confidence interval indicates that there is an 
effect due to C0 2 pressure:The lower C0 2 pressure produces longer times between 
heartbeats. 

Some caution must be exercised in our interpretation of the results because the 
trials with halothane must follow those without. The apparent H-effect may be due 
to a time trend. (Ideally, the time order of all treatments should be determined at 
random.) ■ 


The test in (6-16) is appropriate when the covariance matrix, Cov(X) = 2, 
cannot be assumed to have any special structure. If it is reasonable to assume that 2 
has a particular structure, tests designed with this structure in mind have higher 
power than the one in (6-16). (For 2 with the equal correlation structure (8-14), see 
a discussion of the “randomized block” design in [17] or [22].) 




284 Chapter 6 Comparisons of Several Multivariate Means 


6.3 Comparing Mean Vectors from Two Populations 

A r 2 -statistic for testing the equality of vector means from two multivariate popula¬ 
tions can be developed by analogy with the univariate procedure. (See [11] for a dis¬ 
cussion of the univariate case.) This T 2 -statistic is appropriate for comparing 
responses from one*set of experimental settings (population 1) with independent re¬ 
sponses from another set of experimental settings (population 2). The comparison 
can be made without explicitly controlling for unit-to-unit variability, as in the 
paired-comparison case. 

If possible, the experimental units should be randomly assigned to the sets of 
experimental conditions. Randomization will, to some extent, mitigate the effect 
of unit-to-unit variability in a subsequent comparison of treatments. Although some 
precision is lost relative to paired comparisons, the inferences in the two-population 
case are, ordinarily, applicable to a more general collection of experimental units 
simply because unit homogeneity is not required. 

• Consider a random sample of size from population 1 and a sample of 
size n 2 from population 2. The observations on p variables can be arranged as 
follows: 


Sample 


Summary statistics 


(Population 1) 

1 * 

1 n * 


XH.XU.-'-.Xln, 

x ‘' 

S,- 2(*iy x i) (*iy 

n l 1 /=1 

- x l)' 

(Population 2) 

X 21, x 22> • • • » x 2n 2 

\ n 2 

S 2 = .2 (*2 i * 2 ) (* 2 1 

n 2 “ 1 7=1 

- x 2 )' 


In this notation, the first subscript—1 or 2—denotes the population. 

We want to make inferences about 

(mean vector of population 1) - (mean vector of population 2) = pi i - /t 2 . 

For instance, we shall want to answer the question, Is m i = /* 2 (or, equivalently, is 
Mi ” M 2 = 0)? Also, if Mi “ M 2 * 0, which component means are different? 

With a few tentative assumptions, we are able to provide answers to these questions. 


Assumptions Concerning the Structure of the Data 

1. The sample Xu, Xi 2 ,..., Xi n] , is a random sample of size n 1 from a p-variate 
population with mean vector /*i and covariance matrix X \. 

2. The sample X 2 i, X 22 ,..., X 2 „ 2 , is a random sample of size n 2 from a p-variate 
population with mean vector /t 2 and covariance matrix X 2 - 

3. Also, Xi 1 , X 12 , ■ • •, Xi n] , are independent of X 21 , X 22 , ■.., X 2 „ 2 . (6-19) 

We shall see later that, for large samples, this structure is sufficient for making 
inferences about the p X 1 vector /x 1 - /t 2 . However,when the sample sizes n x and 
n 2 are small, more assumptions are needed. 
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Further Assumptions When n, and n 2 Are Small 

1. Both populations are multivariate normal. 

2. Also, Si = X 2 (same covariance matrix). (6-20) 

The second assumption, that X\ = S 2 , is much stronger than its univariate counter¬ 
part. Here we are assuming that several pairs of variances and covariances are 
nearly equal. 

n \ 

When Si = X 2 = S, 2 ( x i j ~~ x i)( x i/ ~~ x i) ; is an estimate of (n Y - 1)S and 


2( x 2 j ~ * 2 ) ( x 2 j ~ *2)' is an estimate of (n 2 - 1)S. Consequently, we can pool the 

/ =1 

information in both samples in order to estimate the common covariance S. 

We set 

”1 _ "2 

2 (*iy - - * 1 )' + 2 (*2 j - *2)(*2y “ * 2 )' 

c _ Jzl _ !zL _ 

^ P °° ,ed „ . „ O 

ti\ n 2 -- z 

Aii — 1 n 7 — 1 

= —:- -Z Si + —X-- s 2 (6-21) 

n L + n 2 — 2 ni + n 2 — 2 

i _ , " 2 

Since 2j ( x i j ~ x i)( x iy “ x i) has n Y - 1 d.f. and ^ ( x 2 j ~ x 2 )( x 2 / “ x 2 )' has 
7=1 y=l 

n 2 ~ 1 d.f., the divisor (/zj - 1) + ( n 2 - 1) in (6-21) is obtained by combining the 
two component degrees of freedom. [See (4-24).] Additional support for the pool¬ 
ing procedure comes from consideration of the multivariate normal likelihood. (See 
Exercise 6.11.) 

To test the hypothesis that Mi “ M 2 = 5 0 , a s P ec ifi e d vector, we consider the 
squared statistical distance from Xi - x 2 to 8 0 . Now, 

E(Xi - X 2 ) = E(X x ) - E(X 2 ) = mi - M 2 

Since the independence assumption in (6-19) implies that X 1 and X 2 are indepen¬ 
dent and thus Cov(X!, X 2 ) = 0 (see Result 4.5), by (3-9), it follows that 

Cov(X, - X 2 ) = Cov(Xj) + Cov(X 2 ) = ^2 + —X = (— + (6-22) 


n 1 n 2 


Because S poo j ed estimates X, we see that 


\m + /z 2 j Spoo,ed 

is an estimator of Cov(X! - X 2 ). 

The likelihood ratio test of 

H 0 : Mi ~ M 2 ~ 

is based on the square of the statistical distance, T 2 , and is given by (see [1]). 
Reject H 0 if 


T 2 = (xi - *2 - S o y (T + TJs pooIed 


(x, - x 2 - S 0 ) > c 2 (6-23) 
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where the critical distance c 2 is determined from the distribution of the two-samp] e 
T 2 -statistic. 


Result 6.2. If Xu, Xi 2 » • • •, X 1#I1 is a random sample of size n x from N p (it x , 2) and 
X 2 1 , X 22 , * • ■ 1 X 2 m 2 is an independent random sample of size n 2 from N p {ft 2j 2), ther 

T 2 = [X, - X 2 - (Ml - Mi)]' (i; + ^) S P°oied [Xi - x 2 - (m. - M 2 )] - 


is distributed as 


Consequently, 


P (X, - X 2 - (Mt - Pi)) 


(«i + «2 - 2 )p r 

(m + « 2 - p -1) p 


— + —)s pooied 1 (X, - X 2 - (Ml - M 2 » s c 2 = 1 - « 

n j 1 J J 


(«l + « 2 ~ 2 )p 


(m + n 2 - p ~ 1) p 




Proof. We first note that 


Xi - X 2 = —X n 


is distributed as 


-x 12 + ••• + —x Ul —x 2 


n 2 ‘ n 2 


-X 22 -X, 


N p{n-i*> {^ + V 2 ) X 


by Result 4.8, with c x = c 2 — *** — c ;7| — 1/flj and ^Oj + l ~ £/i]+2 ^ni+/i2 

-l/n 2 < According to (4-23), 

(m - l)Si is distributed as W„ J _ 1 (2) and (n 2 - 1)S 2 as W„ 2 _i(2) 

By assumption, the X^-’s and the X 2 /s are independent, so (n x — l)Si and 
(n 2 - l)S 2 are also independent. From (4-24), (ni — 1)S 1 + (n 2 — 1 )S 2 is then dis¬ 
tributed as W ni+na - 2 (S). Therefore, 


T 2 = [ -1 + i-] V2 ( Xl - X 2 - (Ml - #* 2 )) f Spooled 
V"1 n l) 


1 1 V 1 ^ _ _ 

- + —J (Xi - X 2 - (Ml - M2)) 
*1 "2/ 


multivariate normal V ( Wishart random matrix^ 1 f multivariate normal 


random vector 


random vector 


= N p (0, X)' 


. r w„, + n 2 _ 2 (X) - 

+ n 2 — 2_ 


%(0,2) 


which is the ^-distribution specified in (5-8), with n replaced by ny + n 2 - 1. [See 
(5-5) for the relation to F.] 1 
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We are primarily interested in confidence regions for Mi ~ M 2 * From (6-24), we 
conclude that all Mi ~ M 2 within squared statistical distance c 2 of x x - x 2 constitute 
the confidence region. This region is an ellipsoid centered at the observed difference 
Xi - x 2 and whose axes are determined by the eigenvalues and eigenvectors of 
Spooled (^ Spiled)- 


Example 6.3 (Constructing a confidence region for the difference of two mean vectors) 

Fifty bars of soap are manufactured in each of two ways. Two characteristics, 
Xy = lather and X 2 = mildness, are measured. The summary statistics for bars 
produced by methods 1 and 2 are 


*8.3 

L 41 


Obtain a 95% confidence region for Mi — M 2 * 

We first note that S* and S 2 are approximately equal, so that it is reasonable to 
pool them. Hence, from (6-21), 


- 49 r 

Spooled qg Sl + 9 g S 2 ~ 1 5 


*1 “ x 2 


so the confidence ellipse is centered at [ — 1.9, .2]'. The eigenvalues and eigenvectors 
of S pooJe d are obtained from the equation 


0 “ | Spooled AI| — 


2 — A 1 
1 5 - A 


— A 2 — 7A + 9 


so A = (7 i V49 - 36)/2. Consequently, A x = 5.303 and A 2 = 1.697, and the 
corresponding eigenvectors, ej and e 2 , determined from 


ie t - = Aje { , i ~ 1,2 


.290 
&1 “ [_.957_ 


and e 2 = 


By Result 6.2, 


— 

n x n 2 


1 . n(98)(2) 


V «1 n 2 )~ \50 50y (97) 

since F 2i 97(.05) — 3.1. The confidence ellipse extends 

^ M * W - ^ ^ 


Fi.<n(.05) = .25 
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units along the eigenvector e f , or 1.15 units in the e x direction and .65 units in the e 2 
direction. The 95% confidence ellipse is shown in Figure 6.1. Clearly, Mi - fi 2 = 0 
is not in the ellipse, and we conclude that the two methods of manufacturing soap 
produce different results. It appears as if the two processes produce bars of soap 
with about the same mildness (X 2 ), but those from the second process have more 
lather (* 1 ). * m 

Simultaneous Confidence Intervals 

It is possible to derive simultaneous confidence intervals for the components of the 
vector mi — M 2 These confidence intervals are developed from a consideration of 
all possible linear combinations of the differences in the mean vectors. It is assumed 
that the parent multivariate populations are normal with a common covariance 2. 

Result 6.3. Let c 2 = [(«, + n 2 - 2 )p/(n x + n 2 ~ p - 1 With 

probability 1 — a. 

a (Xi ^ X 2 ) ± C Spooled 

will cover — M 2 ) for all a. In particular (i u - M 2 / will be covered by 

(X u - X 2i ) ±c^j^- + ^ Spooled for/ = 1,2,..., p 

Proof. Consider univariate linear combinations of the observations 

Xn,X 12> ...,X lni and Xji,X 22 ..-.,X 2 ,, 2 

given by a'X 1; = a x X w + a 2 X ly2 + ■■■ + a p X ljp and a'X 2; = a x X 2n + a 2 X 2j2 
+ - + a p X 2jp . These linear combinations have sample means and covariances 
a'X 1? a'S^ and a'X 2 , a'S 2 a, respectively, where X lf S l5 and X 2 , S 2 are the mean 
and covariance statistics for the two original samples. (See Result 3.5.) When both 
parent populations have the same covariance matrix, s 2 a = a'Sja and s 2 ^ — a'S 2 a 
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are both estimators of a'Xa, the common population variance of the linear combi¬ 
nations a'Xj and a'X 2 . Pooling these estimators, we obtain 

2 _ («1 ~ + («2 ~ J )« 2 .. 

Vpo°led (#fi + n 2 - 2) 


, *i~l „ n 2 1 

- a' ---S, + ---S. 

n x + n 2 - 2 + * 2 - 2 A 


a Spooled® 

To test H 0 : a '(mi - /i 2 ) = a'5 0 , on the basis of the a'X^ and a'X 2 y, we can form 
the square of the univariate two-sample /-statistic 

,2 _ ["'(S, - X 2 ) - a'( Ml - M2 )] 2 = [a'CX, - X 2 - (mi ~ M2))] 2 , 


— —I 2 

n, + n 2 / ' s *’ pooled 


/ 1 1 
a — + — 

V*1 *2 


According to the maximization lemma with d = (Xj - X 2 - (mj - M 2 )) and 
B = (i/m + l/* 2 )Spooied in (2-50), 


tl < (X! - x 2 (Ml - M 2 ))' ^ + ^Spooled (X] - X 2 - (Mi - M 2 )) 

= T 2 

for all a ^ 0. Thus, 

~ a) = P[T 2 < c 2 ] ~ P[t 2 < c 2 , for all a] 

“ P «'( x i - x 2 ) - a # (Mi “ M2) - c + ^Spooleda for all a 

where c 2 is selected according to Result 6.2. ■ 

Remark. For testing tfo : Mi “ M 2 = 0, the linear combination a'(x! - x 2 ), with 
coefficient vector a oc SJ^oic^Xj - x 2 ), quantifies the largest population difference. 
That is, if T 2 rejects H 0y then a'(xj — x 2 ) will have a nonzero mean. Frequently, we 
try to interpret the components of this linear combination for both subject matter 
and statistical importance. 


(X] - X 2 - (mi ~ M2)) 


Example 6.4 (Calculating simultaneous confidence intervals for the differences in 
mean components) Samples of sizes n x = 45 and n 2 = 55 were taken of Wisconsin 
homeowners with and without air conditioning, respectively. (Data courtesy of Sta¬ 
tistical Laboratory, University of Wisconsin.) Two measurements of electrical usage 
(in kilowatt hours) were considered. The first is a measure of total on-peak consump¬ 
tion (X,) during July, and the second is a measure of total off-peak consumption 
( X 2 ) during July. The resulting summary statistics are 


13825.3 23823.4 

23823.4 73107.4_ ’ 

8632.0 19616.7™ 
19616.7 55964.5 ’ 


n,\ = 45 


n 2 = 55 
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(The off-peak consumption is higher than the on-peak consumption because there 
are more off-peak hours in a month.) 

Let us find 95% simultaneous confidence intervals for the differences in the 
mean components. 

Although there appears to be somewhat of a discrepancy in the sample vari¬ 
ances, for illustrative purposes we proceed to a calculation of the pooled sample co- 
variance matrix. Here 


"i - 1 < 

n x + n 2 ~ 2 


n 2 - 1 


'10963.7 

21505.5 


7 21505.5 
5 63661.3 


2 (”l + ”2 ~ 2 )P e. / \ 

/li + tin — p — 1 


1 + /*2 p 1 

= (2.02) (3.1) = 6.26 


■^2,97(*05) 


With - M 2 = [Mu ~ M 2 i»Mi 2 ~ M 22 L the 95% simultaneous confidence inter¬ 
vals for the population differences are 

M 11 ~ M2i : (204.4 - 130.0) ± V 626 109637 


21.7 SftrftiS 127.1 


(o/i-peak) 


- ju 22 : (556.6 - 355.0) ± V?(26 63661.3 


74.7 — f ±\2 m 22 — 328.5 


(o/jf-peak) 


We conclude that there is a difference in electrical consumption between those with 
air-conditioning and those without. This difference is evident in both on-peak and 
off-peak consumption. 

The 95% confidence ellipse for yui “ M 2 is determined from the eigenvalue- 
eigenvector pairs Aj = 71323.5, e'i = [.336, .942] and A 2 = 3301.5, e 2 = [.942, —.336]. 

Since 


l- + l.y = vmm^ 5 + j- 5 


626 = 134.3 


1 1 

— H- 

n x n 2 


T 5 + h) 626 = 28 - 9 


we obtain the 95% confidence ellipse for /xj - /x 2 sketched in Figure 6.2 on page 291 
Because the confidence ellipse for the difference in means does not cover 0' = [0,0] 
the r 2 -statistic will reject Hq\ Mi — M 2 — 0 at the 5% level. 
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^ 12-^22 



The coefficient vector for the linear combination most responsible for rejection 
is proportional to Spooled(*i “ * 2 )- (See Exercise 6.7.) ■ 

The Bonferroni 100(1 - a)% simultaneous confidence intervals for the p popu¬ 
lation mean differences are 



where t n ^ ni ^ 2 {a/2p) is the upper 100(a/2p)th percentile of a f-distribution with 
n \ + n 2 ~ 2 d.f. 

The Two-Sample Situation When 2, # X 2 

When * 2 2 . we are unable to find a “distance” measure like T 7 , whose distribu- 
tion does not depend on the unknowns Xj and X 2 * Bartlett’s test [3] is used to test 
the equality of Xi and X 2 in terms of generalized variances. Unfortunately, the con¬ 
clusions can be seriously misleading when the populations are nonnormal. Nonnor¬ 
mality and unequal covariances cannot be separated with Bartlett’s test. (See also 
Section 6.6.) A method of testing the equality of two covariance matrices that is less 
sensitive to the assumption of multivariate normality has been proposed by Tiku 
and Balakrishnan [23]. However, more practical experience is needed with this test 
before we can recommend it unconditionally. 

We suggest, without much factual support, that any discrepancy of the order 
<r\ n ~ 4 <r 2i i/, or vice versa, is probably serious. This is true in the univariate case. 
The size of the discrepancies that are critical in the multivariate situation probably 
depends, to a large extent, on the number of variables p. 

A transformation may improve things when the marginal variances are quite 
different. However, for n 7 and n 2 large, we can avoid the complexities due to 
unequal covariance matrices. 
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Result 6.4. Let the sample sizes be such that n { - pand/ 2 2 - pare large. Then, an 3 
approximate 100(1 a) % confidence ellipsoid for/i, - /t 2 is given by all/i, - 1 

satisfying §9 

[x, - X 2 - (/*, - M 2 )]' ^ Si + ~ S 2 J [x, - x 2 - (Ml - M 2 )] xl(a) "^g 

1 

where x 2 p( a ) i s th e upper (100ar)th percentile of a chi-square distribution with p d.f. :jg 
Also, 100(1 - a)% simultaneous confidence intervals for all linear combinations 
a'(/*i - m 2 ) are provided by l| 

a'(Mi - M 2 ) belongs to a’(x, - x 2 ) ± V^(a) ^Ja'C~Si + ~S 2 ja 


Proof. From (6-22) and (3-9), 


£(Xi - X 2 ) = Mi - M2 


Cov(X, - X 2 ) = Cov(X,) + Cov(X 2 ) = ^-2, + ^1 2 

By the central limit theorem, X] - X 2 is nearly N p [fi\ - /i 2 ,/ij^X x If Sj 

and X 2 were known, the square of the statistical distance from Xi ~ X 2 to - fi 2 
would be 

[X, - X 2 - (Mi - M 2 )]' 1 [X, - X 2 - (m, - M 2 )] 

This squared distance has an approximate ^-distribution, by Result 4.7. When n { and 
n 2 are large, with high probability, S] will be close to Xj and S 2 will be close to X 2 . 
Consequently, the approximation holds with S x and S 2 in place of X x and X 2 , 
respectively. 

The results concerning the simultaneous confidence intervals follow from 
Result 5 A.1. ■ 


Remark. lfn x = n 2 = / 2 ,then (n - 1 )/{n + n - 2) = 1/2, so 

m 1 n 2 2 n v n + n - 2 \n n 


With equal sample sizes, the large sample procedure is essentially the same as the 
procedure based on the pooled covariance matrix. (See Result 6.2.) In one dimen¬ 
sion, it is well known that the effect of unequal variances is least when n x = n 2 and 
greatest when n x is much less than n 2 or vice versa. 
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Example 6.5 (Large sample procedures for inferences about the difference in means) 

We shall analyze the electrical-consumption data discussed in Example 6.4 using the 
large sample approach. We first calculate 


1 13825.3 23823.4 1 8632.0 ] 9616.7 


45 23823.4 73107.4 J 55 ^19616.7 55964.5 

464.17 886.08 ~ 

886.08 2642.15 


The 95% simultaneous confidence intervals for the linear combinations 


a'(Mi ~ M 2 ) = [1,0] 


Mil “ M 21 
Mi 2 “ M 22 


Mil ~ M 21 


a'(Mi ~ M 2 ) = [0,1] 


Mil “ M 21 
Mi 2 - M 22 


— Ml2 M 22 


are (see Result 6.4) 

M 11 -M 2 C 74.4 ± V5$9 V464A7 or (21.7,127.1) 

M 12 ~ M 22 * 201.6 ± V529 V2642.15 or (75.8, 327.4) 

Notice that these intervals differ negligibly from the intervals in Example 6.4, where 
the pooling procedure was employed. The 7" 2 -statistic for testing H 0 : — /i 2 = 0 is 


T 2 = [x t - x 2 ]' -S, 
n 1 


204.4 - 130.0 ' 464.17 886.08 _1 204.4 - 130.0 
556.6 - 355.0 886.08 2642.15 556.6 - 355.0 


= [74.4 201.6] (10" 


59.874 -20.080 74.4 

-20.080 10.519 201.6 


= 15.66 


For a = .05, the critical value is * 2 (.05) = 5.99 and, since T 2 = 15.66 > * 2 (-ft>) 
= 5.99, we reject H 0 . 

The most critical linear combination leading to the rejection of H 0 has coeffi¬ 
cient vector 


■ 0C f^-S 1 +“ s 2 ) (xi “ x 2 ) = (10' 4 ) 


59.874 -20.080 74.4 

-20.080 10.519 201.6 


The difference in off-peak electrical consumption between those with air condi¬ 
tioning and those without contributes more than the corresponding difference in 
0 / 2 -peak consumption to the rejection of H 0 : Mi “ M 2 “ 0. ■ 
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A statistic similar to T 2 that is less sensitive to outlying observations for small 
and moderately sized samples has been developed byTiku and Singh [24], However 
if the sample size is moderate to large, Hotelling’s T 2 is remarkably unaffected bv- 
slight departures from normality and/or the presence of a few outliers. 

An Approximation to the Distribution of T 2 for Normal 
Populations When Sample Sizes Are Not Large 

One can test H 0 : mi - /i 2 = 0 when the population covariance matrices are un¬ 
equal even if the two sample sizes are not large, provided the two populations are 
multivariate normal. This situation is often called the multivariate Behrens-Fisher 
problem. The result requires that both sample sizes n x and n 2 are greater than p, the 
number of variables. The approach depends on an approximation to the distribution 
of the statistic 


T 2 = (X! - X 2 - (mi ~ M2))|-Si + “S 2 j (Xj - X 2 - ( Ml - M 2 )) (6-27j 

which is identical to the large sample statistic in Result 6.4. However, instead of 
using the chi-square approximation to obtain the critical value for testing H 0 the 
recommended approximation for smaller samples (see [15] and [19]) is given by 

7,2 = v _ p + 1 7 >.v-p +1 (6-28) 

where the degrees of freedom v are estimated from the sample covariance matrices 
using the relation 


Y 2 -- 


"[(Mi; 8 ' 


/ \u\ 




where min(/z b n 2 ) ^ v ^ n x + n 2 . This approximation reduces to the usual Welch 
solution to the Behrens-Fisher problem in the univariate (p = 1) case. 

With moderate sample sizes and two normal populations, the approximate level 
a test for equality of means rejects H 0 : Mi “ M 2 = 0 if 


( x i ~ x 2 - (m1 ~ M2))' ~ _ S 1 

im 


(x 3 - x 2 - (mi “ M 2 )) > 


V - p + 


1 Fp,v-p+ i( a ) 


where the degrees of freedom v are given by (6-29). This procedure is consistent 

with the large samples procedure in Result 6.4 except that the critical value xl( a ) ls 

vp p 

replaced by the larger constant — ^ + ^ F p v _ p+1 (a). 

Similarly, the approximate 100(1 - a)% confidence region is given by all 
Mi - M 2 such that 

(xj - x 2 - (mi - M2))' —Si + — S 2 (xj - x 2 - (mi - M2)) - “- P 1 F p ,v-p+i( a 

n\ n 2 J v — p 4- 1 y y 
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For normal populations, the approximation to the distribution of T 2 given by 
(6-28) and (6-29) usually gives reasonable results. 


Example 6.6 (The approximate T 2 distribution when X, * S 2 ) Although the sample 
sizes are rather large for the electrical consumption data in Example 6.4, we use 
these data and the calculations in Example 6.5 to illustrate the computations leading 
to the approximate distribution of T 2 when the population covariance matrices are 
unequal. 

We first calculate 


1 

s, 

1 

'13825.2 

23823.4' 


'307.227 

529.409' 

"1 


45 

.23823.4 

73107.4. 


.529.409 

1624.609. 


s 2 

J_ 

r 8632.0 

19616.7' 


'156.945 

356.667' 

"2 


55 

.19616.7 

55964.5. 


356.667 

1017.536. 


and using a result from Example 6.5, 


is I+ ±s 2 r=(io-<)f 59 - 874 - 2oo8 ° 

"1 "2 J “20.080 10.519 


Consequently, 


1 * \ 1 « 11 c r 

ni |_*i n 2 2 \ 


307.227 529.409] ^ T 59.874 -20.080 
529.409 1624.609 J [-20.080 10.519 


1 „ 1 e A 1 c 

3i -I-3 2 

"1 |ni n 2 


Further, 


iji c lj- 1 

„ ^2 hj -I-S 2 

n 2 L W 1 «2 


Lc 1 *Y _ F - 77 6 -.060]r .776 -.0601 


156.945 356.667 

356.667 1017.536 


.776 

-.060 

-.092 

.646. 

.608 

-.085' 

-.131 

.423. 

'.224 

-.060' 

..092 

.354, 


1 c 1 c ^ 1 c TY 

*2 —a> -f-a-> 

"2 L«1 »2 J / 


.224 .060][ .224 .060] _ [.055 .035 
-.092 .354JL—-092 .354J ~ [.053 .131 
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Then 





1 f 1 

Y i / 

' 1 


1 ^ 


( SJ 

— Si 

+ 

—s> 

«i l 1 

[\ n l V 



n 2 J 

1 / 1 


/ 1 c 


1 „ 

—Y tr 

( "S 21 

— Si 

+ 

— S 

«2 l 1 

A«2 ' 

,"1 


n 2 


^ {(.608 + .423) + (.776 + ,646) 2 } = .067 

+ + A)"'])'} 



= -^{(.055 + .131) + (.224 + ,354) 2 } = 



Using (6-29), the estimated degrees of freedom v is 



2 + 2 2 

V ~ .0678 + .0095 " 77 6 
and the a = .05 critical value is 

vp r 77.6 X 2 ....155.2 

w _, + 1 *Wi<-°5) “ 77.6-2 + 1 F2J76 - 2+i(05) “ 76.6 312 ~ 632 ^ 

, • ^ 

From Example 6.5, the observed value of the test statistic is T = 15.66 so the^ 
hypothesis // 0 : Mi - M 2 = ® is rejected at the 5% level. This is the same conclusion^ 
reached with the large sample procedure described in Example 6.5. apj 

As was the case in Example 6.6, the F PtV - p +\ distribution can be defined wifflp 
noninteger degrees of freedom. A slightly more conservative approach is to use theHg 
integer part of v. 


6.4 Comparing Several Multivariate Population Means 
(One-Way M A NOVA) 

Often, more than two populations need to be compared. Random samples, collected; 
from each of g populations, are arranged as 


Population 1: 

Xu, X 12 ,. 


Population 2: 

X 2 ], X 22 , ■ 

...X 2 „ 2 

Population g: 

X gl ,x, 2 ,. 

Y 


MANOVA is used first to investigate whether the population mean vectors are the 
same and, if not, which mean components differ significantly. j 

Assumptions about the Structure of the Data for One-Way MANOVA 

,Jp 

L X € j , X f2 ,. •., , is a random sample of size /if from a population with mean fiq 

€ = 1,2,..., g. The random samples from different populations are independen| 
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2. All populations have a common covariance matrix 2. 

3. Each population is multivariate normal. 

Condition 3 can be relaxed by appealing to the central limit theorem (Result 4.13) 
when the sample sizes n e are large. 

A review of the univariate analysis of variance (ANOVA) will facilitate our 
discussion of the multivariate assumptions and solution methods. 


A Summary of Univariate ANOVA 

In the univariate situation, the assumptions are that X ei , Xe2>-' ■ , X e „ e is a random 
sample from an a 2 ) population, f = 1, 2,..., g t and that the random samples 

are independent. Although the null hypothesis of equality of means could be formu¬ 
lated as mi — M 2 = * * * = Mg> it is customary to regard as the sum of an overall 
mean component, such as m, and a component due to the specific population. For 
instance, we can write Me “ M + (mc — m) or Me = M + T e where T e = Me “ M- 
Populations usually correspond to different sets of experimental conditions, and 
therefore, it is convenient to investigate the deviations t € associated with the Tth 
population (treatment). 

The reparameterization 

Me = M + T e 

f th population | f overall ] / f th population 

mean J l mean ) u treatment) effect 



leads to a restatement of the hypothesis of equality of means. The null hypothesis 
becomes 

H 0 : ti = t 2 = ‘' • = T g = 0 

The response X< } -, distributed as N(p + r e , or 2 ), can be expressed in the suggestive 
form 


(overall mean) 


( treatment 
effect 


e ej 

( random) ( 6 ‘ 33 ) 
V error J 


where the e C j are independent JV(0, a 2 ) random variables. To define uniquely 
the model parameters and their least squares estimates, it is customary to impose the 

constraint ^ - 0. 

c=\ 

Motivated by the decomposition in (6-33), the analysis of variance is based 
upon an analogous decomposition of the observations, 

x ej = X + (x € -x) + (x (j - x e ) 

(observation) f OV , era11 ) f, ) (residual) ^ 

^sample mean J y treatment effect^ v } 


where jc is an estimate of m, = ( x e "" x ) is an estimate of t € , and (x e j - X{) is an 
estimate of the error e V] . 
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Example 6.7 (The sum of squares decomposition for univariate ANOVA) Consider 
the following independent samples. 

Population 1: 9,6,9 
Population 2: 0,2 
Population 3: 3,1,2 

Since, for example, = (3 + 1 + 2)/3 = 2 and x = (9 + 6 + 9 + 0 + 2 + 

3 + 1 + 2)/8 = 4 , we find that 

3 = jjj = X + (i 3 - *) + (*31 - * 3 ) 

= 4 + (2 - 4) + (3 - 2) 

= 4 + (-2) + 1 

Repeating this operation for each observation, we obtain the arrays 

/9 4 )\ (1 4 4 'l / 4 4 “ I (' ~ 2 *' 

(;; J * (:: J + U" J 1".»J 

U - mean + treatment effect + residual 

observation - medI1 / v _ T \ 

, , fs) (x<-x) (x (j xt) 


observation — mean 

(x Cl ) & 


The Question of equality of means is answered by assessing whether the 
contribution of the treatment array is large relative to the residuals. (Our esti- 

mates f € = *e ~ * of r < alwayS satisfy = °* Under H o» each Te 1S an 

estimate of zero.) If the treatment contribution is large, H 0 should be rejected. The 
size of an array is quantified by stringing the rows of the array out into a vector and 
i l „n» it. cmiared length. This quantity is called the sum of squares (SS). For 
■- id the Vector / . |9. 6 .9,0.2,3,1.2], It, squared 

length is 

S^.#*#.**#.**?* 1 - + 2-.216 

Similarly, 

CC = a 2 + 42 + 42 + 42 + 4 2 + 4 2 + 4 2 + 4 2 = 8(4 2 ) = 128 

” I 4 r + 4= + 4= + (-3)9 * (-3)9 + (-2)9 * (-2)9 . (-2)9 

tr = 3 (4 2 ) + 2(-3 ) 2 + 3(—2 ) 2 = 78 
and the residual sum of squares is 

SSrw = 1 2 + (-2 ) 2 + l 2 + (- 1) 2 + l 2 + l 2 + (- 1) 2 + 0 2 = 10 
m sums of squares satisfy the same decomposition, (6-34), as the observations 
Consequently, 

SS obs = SS mean + SS tr + SS res 

rhroifi = 128 + 78 + 10. Hie breakup into sums of squares apportions variability in 

the combined samples into mean, treatment and residual (error) components .to 
analysis of variance proceeds by comparmg the relative sizes of SS tr and SS res . If H 0 
fs true variances computed from SS tr and SS re5 should be approximately equal. ■ 
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The sum of squares decomposition illustrated numerically in Example 6.7 is so 
basic that the algebraic equivalent will now be developed. 

Subtracting x from both sides of (6-34) and squaring gives 

(x Cj - xf = (x e ~ xf + (x cj - xt) 1 + 2(x € - x)(x e/ - x e ) 

We can sum both sides over j\ note that 2 ( x ej ~ x t) ~ 0, and obtain 

7 = 1 

]£ (x € j - xf = n e (x e - xf + ^ (xgj - x e f 
7=1 7=1 

Next, summing both sides over € we get 

xt ( x *i ~ *) 2 = 2 ~ xf + 2 2 ( x (j ~ *r) 2 ( 6 ‘ 35 ) 

f=l j =1 r=i f=l y=l 


total (corrected) SS J ^between (samples) SS ) ^within (samples) SS 


EX4 j = (/»1 + n 2 + ••• + n g )X 2 + 2 n c {x ( - xf + 2 ^ (*0 ~ x tf 

C = 1 7 = 1 £=1 ^=1 7=1 

(SS obs ) = (SS niean ) + (SS tr ) + (SS res ) (6-36) 


In the course of establishing (6-36), we have verified that the arrays represent¬ 
ing the mean, treatment effects, and residuals are orthogonal. That is, these arrays, 
considered as vectors, are perpendicular whatever the observation vector 
y' = [x n ,..., x 1/fl , x 2 i,..., X 2 „ r ■ ■ - 7 x gnJ- Consequently, we could obtain SS res by 
subtraction, without having to calculate the individual residuals, because SS res = 
SSobs “ SS mean - SS tr . However, this is false economy because plots of the residu¬ 
als provide checks on the assumptions of the model. 

The vector representations of the arrays involved in the decomposition (6-34) 
also have geometric interpretations that provide the degrees of freedom. For an ar¬ 
bitrary set of observations, let [jc u ,..., x lrtl , x 2i , ..., x lnv ..., x gn ] = y\ The ob¬ 
servation vector y can lie anywhere in n = n y + n 2 + • • • + n g dimensions; the 
mean vector x 1 = [ x ,..., x] must lie along the equiangular line of 1 , and the treat¬ 
ment effect vector 



= (x, - x)u, + (x 2 - x)u 2 + •■■+(*,- 
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lies in the hyperplane of linear combinations of the g vectors u 1) u 2 ,...,u g . Since i 
1 = Ul + U 2 + ••' + the mean vector also lies in this hyperplane, and it is s 
always perpendicular to the treatment vector. (See Exercise 6.10.) Thus, the mean 1 
vector has the freedom to lie anywhere along the one-dimensional equiangular line, 
and the treatment vector has the freedom to lie anywhere in the other g - 1 di¬ 
mensions. The residual vector,e = y - (xl) — [(*1 - x)ii] + • • • + (x g — x)u g ] is 
perpendicular to both the mean vector and the treatment effect vector and has the 
freedom to lie anywhere in the subspace of dimension n - (g - l)-~ 1 = n ~ g~ 
that is perpendicular to their hyperplane. 

To summarize, we attribute 1 d.f. to SS mca n. g _ . 1 d.f. to SS tr , and n - g = ■ 
(n, + n 2 + •• ■ + n g ) - g d.f. to SS re5 . The total number of degrees of freedom is 
n L + n 2 + • - + n g . Alternatively, by appealing to the univariate distribution 
theory! we find that these are the degrees of freedom for the chi-square distributions 
associated with the corresponding sums of squares. 

The calculations of the sums of squares and the associated degrees of freedom 
are conveniently summarized by an ANOVA table. 


ANOVA Table for Comparing Univariate Population Means 


Source 
of variation 

■r*-o * 

Sum of squares (SS) 

Degrees of 
freedom (d.f.) 

Treatments 

SS„ = n e (X( - xf 

e=i 

SS res = i X (xt, - Xef 
y-i 

8 “ 1 

Residual 

(error) 

X n ( ~ 8 

e=i 

Total (corrected 
for the mean) 

SScor = ^ X (*0 ~ S ) 2 

( = 1 )~l 

±n ( -l 


The usual F-test rejects Ho- Ti - r 2 — ■ • ■ - f g - 0 at level a if 


SS J(g - 1) 
SS res / (X"r - s 


Fg-\,Zn t -g( a ) 


where F _i z n -gM is the u PP er ( 100a ) th percentile of the F-distribution with 
p - 1 and Sn/ - g degrees of freedom. This is equivalent to rejecting H 0 for 
large values of SS tr /SS res or for large values of 1 + SS tr /SS res . The statistic 
appropriate for a multivariate generalization rejects H 0 for small values of the 
reciprocal 


1 + SS tr /SS r{ 


SS res 

SS res + ss lr 
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Example 6.8 (A univariate ANOVA table and F-test for treatment effects) Using the 
information in Example 6.7, we have the following ANOVA table: 


Source 
of variation 

Sum of squares 

Degrees of freedom 

Treatments 

SS tr = 78 

g~ 1 =3-1 = 2 

Residual 

SS res = 10 

^ n e ~ g ~ ( 3 + 2 + 3) - 3 = 5 
e=\ 

Total (corrected) 

SS cor = 88 

1>-1=7 

e=i 


Consequently, 

ss lr /(g - 1) , 78/2 
SS t J(Zn e - g) 10/5 

Since F = 19.5 > F 2 , 5 (. 01 ) = 13.27, we reject H 0 : tj = r 2 = r 3 = 0 (no treatment 
effect) at the 1 % level of significance. ■ 

Multivariate Analysis of Variance (MANOVA) 

Paralleling the univariate reparameterization, we specify the MANOVA model: 

MANOVA Model For Comparing g Population Mean Vectors 

Xzy = p + T ( + e (j , j = 1,2,..., n e and € = l,2,...,g (6-38) 

where the e €j - are independent N p (0, X) variables. Here the parameter vector jx 
is an overall mean (level), and r e represents the £th treatment effect with 

X n t T < - °- 

f=i 

According to the model in (6-38), each component of the observation vector Xgj sat¬ 
isfies the univariate model (6-33). The errors for the components of X e j are corre¬ 
lated, but the covariance matrix X is the same for all populations. 

A vector of observations may be decomposed as suggested by the model. Thus, 

x O' 

(observation) 

The decomposition in (6-39) leads to the multivariate analog of the univariate 
sum of squares breakup in (6-35). First we note that the product 

(x (j - x)(x (j - x)' 


X + (Xf ~ x) + (x ej - x e ) 

/overall sample') /residual) (6-39) 

V mean M ) l effect rj V / 
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can be written as 

(x f/ - x)(x t/ - X)' = [( x f/ - St) + - *t) + < 5f " s)] ' 

= (x C y - x c ) (X(j - xe)' + (xtj-*f)(*t " *)' 

+ (x f - x)(*a - *()' + (*< ~ *)(*< ~ x)' 

The sum over j of the middle two expressions is the zero matrix, because 
^ ( Xf; - x f ) = 0 Hence, summing the cross product over i and j yields 

pi 

t S (x t; - x) (*/ - *)' = t " 5 > (5f " 5) ' + | (**/ ' (xf ' " 5() ' ■ 

f=l y-1 7 


of squares and cross 
products 


sum of squares and 
cross products 


of squares and cross 
products 


The within sum of squares and cross products matrix can be expressed as 

g /If 

W = 2 2 («f/ “ Xf) (*</ - Xf)' 

f=! Cl (6-41) 

= (ni - l)Sj + («2 ~ 1)% + f ( n g ~ 

where S is the sample covariance matrix for the €th sample. This matrix is a gener 
alization of the («, + /%- 2)S pooled matrix encountered in the two-sample case. It 

nlavs a dominant role in testing for the presence of treatment effects. 

Analogous to the univariate result, the hypothesis of no treatment effects, 


tf 0 : T i = t 2 


• = T * = 0 


is tested bv considering the relative sizes of the treatment and residual sums of 
sauares and cross products Equivalently, we may consider the relative sizes of the 
residual and total (corrected) sum of squares and cross products Formally, we sum 
marize the calculations leading to the test statistic in a MANO VA table. 

MANOVA Table for Compari ng Population Mean Vectors _ 

Source Matrix of sum of squares and Degrees of 

of v ariation cross products (SSP) _freedomJdl)__ 

Treatment B = t n e (x, - x)(x ( - x)' £-1 

g n r 8 

Residual (Error) W = 2 2 («»/ " if) (»*/ " ^ g "f " * 

f=l /-1 _ 




Total (corrected 
for the mean) 


B + w = ^ 2 (Xf; _ *)(*</ ~ x)' 

<=l (=i 


/If — 1 
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This table is exactly the same form, component by component, as the ANOVA table, 
except that squares of scalars are replaced by their vector counterparts. For exam¬ 
ple, (x e - x ) 2 becomes (x f - x)(x^ - x)'. The degrees of freedom correspond to 
the univariate geometry and also to some multivariate distribution theory involving 
Wishart densities. (See [1].) 

One test of // 0 : T\ = r 2 = * * • = r g - 0 involves generalized variances. We re¬ 
ject H 0 if the ratio of generalized variances 

2 2 (*</ - Xf) (x ei - Xf)' 

£ 

e=\ 

is too small. The quantity A* = |W|/|B + W|, proposed originally by Wilks 
(see [25]), corresponds to the equivalent form (6-37) of the F-test of H Q : no treat¬ 
ment effects in the univariate case. Wilks’ lambda has the virtue of being convenient 
and related to the likelihood ratio criterion. 2 The exact distribution of A* can be 
derived for the special cases listed in Table 6.3. For other cases and large sample 
sizes, a modification of A* due to Bartlett (see [4]) can be used to test // 0 . 



jw| 

B + W| 



2 Wilks’ lambda can also be expressed as a function of the eigenvalues of A], A 2t ..., A* of W ! B as 



where s = min (p, g - 1), the rank of B. Other statistics for checking the equality of several multivari¬ 
ate means, such as Pillai’s statistic, the Lawley-Hotelling statistic, and Roy’s largest root statistic can also 
be written as particular functions of the eigenvalues of For large samples, all of these statistics are, 

essentially equivalent. (See the additional discussion on page 336.) 
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Bartlett (see [4]) has shown that if Hq is true and = n is large, 

( i 0 7 + «)V , t f 1 + ^ 

(" 1 2 ) ln A - ( n 1 2 ) \|B + W| 




has approximately a chi-square distribution with p(g - 1) d.f. Consequently, for 
= n large, we reject H Q at significance level a if 


B + W 


> Xp(g-l)( a ) 


where Xp(g- i)( a ) * s £h e upper (lQOa)th percentile of a chi-square distribution with 
p(g~l)dl 


Example 6.9 (A MANOVA table and Wilks* lambda for testing the equality of three 
mean vectors) Suppose an additional variable is observed along with the variable 
introduced in Example 6.7: The sample sizes are — 3,rc 2 = 2, and n 3 = 3. 
Arranging the observation pairs x €; in rows, we obtain 


withxj — 
andx = 


We have already expressed the observations on the first variable as the sum of an 
overall mean, treatment effect, and residual in our discussion of univariate 
ANOVA. We found that 


9 6 9 
0 2 
3 1 2 


(observation) 

and 


r 4 4 4 
4 4 
K 4 4 4 

(mean) 


4 

4 


/ 1 

-2 


3 

-3 

+ 

-1 

1 


-2 

-2 

- 2 / 

V 1 

-1 

0 ) 


treatment j 
effect j 


(residual) 


SS obs — SS mean + SS tr + SS res 
216 = 128 + 78 + 10 
:ted) = S&*. - SS mMn = 216 - 128 = 88 


Total SS (corrected) = SS obs - SS mean = 216 - 128 - 88 
Repeating this operation for the observations on the second variable, we have 


3 2 7 

4 0 

8 9 7 


(observation) 


5 5 5 
5 5 
,5 5 5 

(mean) 


treatment 

effect 


-1 -2 3 

2 -2 
0 1 -1 

(residual) 
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and 

SS obs — SS mean + SS tr + SS res 
272 - 200 + 48 + 24 

Total SS (corrected) = SS obs - SS mean = 272 - 200 = 72 

These two single-component analyses must be augmented with the sum of entry- 
by-entry cross products in order to complete the entries in the MANOVA table. 
Proceeding row by row in the arrays for the two variables, we obtain the cross 
product contributions: 

Mean: 4(5) + 4(5) + ••■ + 4(5) = 8(4) (5) = 160 
Treatment: 3(4) (-1) + 2(-3)(-3) + 3(-2)(3) = -12 
Residual: 1(-1) + (-2) (-2) + 1(3) + (-1) (2) + • • ■ + 0(-l) = 1 
Total: 9(3) + 6(2) + 9(7) + 0(4) + ••• + 2(7) = 149 
Total (corrected) cross product = total cross product - mean cross product 

= 149 - 160 = -11 

Thus, the MANOVA table takes the following form: 

Source Matrix of sum of squares 

of variation and cross products Degrees of freedom 



Equation (640) is verified by noting that 

88 -111 _ r 78 -12l J10 l" 

-11 72J " [-12 48_j + [_ 1 24_ 


Using (642), we get 

10 

1 



A*- ' W l 

1 

24 

10(24) - (l) 2 

_ _?39 _ 

A |B + W| 

88 

-11 

88(72) - (-11) 2 

~ 6215 -° 385 


-11 

72 
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Since p — 2 and g ~ 3, Table 63 indicates that an exact test (assuming normal¬ 
ity and equal group covariance matrices) of // 0 : rj = r 2 — r 3 = 0 (no treatment 
effects) versus Hp. at least one =£ 0 is available.To carry out the test, we compare 
the test statistic 


1 - Va* \ - g ~ 
Va* / (g - !) 


1 - V\0385 
VC0385 


8-3-1 
3 - 1 


with a percentage point of an ^-distribution having v x = 2(g — 1 ) = 4 and. 
v 2 = 2(ln € - g - 1) = 8 d.f. Since 8.19 > F 48 (.01) = 7.01, we reject H 0 at the 
a = .01 level and conclude that treatment differences exist. m 

When the number of variables, p, is large, the MANOVA table is usually not 
constructed. Still, it is good practice to have the computer print the matrices B and 
W so that especially large entries can be located. Also, the residual vectors 

e 0 = Xf j - \ e 

should be examined for normality and the presence of outliers using the techniques 
discussed in Sections 4.6 and 4.7 of Chapter 4. 

Example 6.10 (A multivariate analysis of Wisconsin nursing home data) The 

Wisconsin Department of Health and Social Services reimburses nursing homes in 
the state for the services provided. The department develops a set of formulas for 
rates for each facility, based on factors such as level of care, mean wage rate, and 
average wage rate in the state. 

Nursing homes can be classified on the basis of ownership (private party, 
nonprofit organization, and government) and certification (skilled nursing facility, 
intermediate care facility, or a fcombination of the two). 

One purpose of a recent study was to investigate the effects of ownership or 
certification (or both) on costa Four costs, computed on a per-patient-day basis and 
measured in hours per patient day, were selected for analysis: X\ = cost of nursing 
labor, X 2 = cost of dietary labor, X 3 ~ cost of plant operation and maintenance labor, 
and X 4 = cost of housekeeping and laundry labor. A total of n = 516 observations 
on each of the p = 4 cost variables were initially separated according to ownership. 
Summary statistics for each of the g = 3 groups are given in the following table. 


£ = 1 (private) 


£ = 2 (nonprofit) 


Number of 
observations 

f\ 1 = 271 


n 2 = 138 


i - 3 (government) n 3 - 107 

S n < = 516 

€=1 


Sample mean vectors 
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Sample covariance matrices 

.291 1 [.561 

-.001 .011 = .011 .025 

Sl " .002 .000 .001 ’ 2 _ .001 .004 .005 

.010 .003 .000 .010J |_.° 37 007 -002 

'.261 

.030 .017 

83 - .003 -.000 .004 

.018 .006 .001 .013_ 

Source: Data courtesy of State of Wisconsin Department of Health and Social"Services. 


Since the S^’s seem to be reasonably compatible, 3 they were pooled [see (6^11)] 
to obtain 


W = (n x - 1)S 2 + (n 2 - 1)S 2 + (n 3 - 1)S 3 

"182.962 

4.408 8.200 

1.695 .633 1.484 

9.581 2.428 .394 6.538 


«lXi + n 2 X 2 + «3*3 

+ «2 + n 3 


B = 2 ”f( x < ~ *)(*f - *)' = 

f=l 


3.475 



1.111 

1.225 


.821 

.453 

.235 

.584 

.610 

.230 


To test Hq\ Tj = r 2 = r 3 (no ownership effects or, equivalently, no difference in av¬ 
erage costs among the three types of owners—private, nonprofit, and government), 
we can use the result in Table 6.3 for g = 3. 

Computer-based calculations give 


A * = Tb + wI = 7714 


3 However, a normal-theory test of H 0 : Si = X 2 = 5 3 would reject H 0 at any reasonable signifi¬ 
cance level because of the large sample sizes (see Example 6.12). 
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Zru - p ~ 2 \ ( 1 ~ VA* 
p ) \ Va* 


516 - 4 - 2\ / 1 - VTm^ = 

4 A vrm ) 


Let a = .01, so that F 2(4 ).2(5tO)(-01) = ^i(-01)/8 = 2.51. Since 17.67 > *Wj)l) * 
2.51, we reject H 0 at the 1% level and conclude thataverage costs differ, depending on 

type of ownership lye ^ (he resu)ts based on this “exact” test with tht»e 

obtained using the large-sample procedure summarized in (6-43) and (6-44). For the 
present example, 2n e = » = 516 is large, and H 0 can be tested at the a = .01 level 

by comparing 

-(« - 1 - 0> ♦ '“(jiTSi) ■ -5U.5ln(.7 7 14, - 132.76 

With V 2 , if 01 ) = yg(-Ol) = 2°.09. Since 132.76 > *£(.01) = 20.09, we reject H 0 
at the 1 % level. This result is consistent with the result based on the foregoing 

F-statistic. 

6.5 Simultaneous Confidence Intervals for Treatment Effects 

When the hypothesis of equal treatment effects is rejected, those effects that led to 
the rejection of the hypothesis are of interest. For pairwise comparisons, the Bon- 
ferroni approach (see Section 5.4) can be used to construct simultaneousconfidence 
intervals for the components of the differences r t - r ( (or /** - #»<)• These inter¬ 
vals are shorter than those obtained for all contrasts, and they require critical values 

only for the univariate f-statistic. ,, -. _— _— 

Let r ki be the ith component of t* . Since r* is estimated by t* - x* x 

*„ = *«-*. 

. - - _ = . _ j is the difference between two independent sample means 

The two-sample f-based confidence interval is valid with an appropriately 
modified a. Notice that 


Var(f*j - T Ci ) - Va r {Xu \ nk + ntj" 

where <r ■ is the ith diagonal element of 2. As suggested by (6-41), Var (X k , - Xu) 
is estimated by dividing the corresponding element of W by its degrees of freedom. 

That is, 

— _ - , (1 l\ w u 

Wu-x (i ) = [- + -) — 

where Wu is the ith diagonal element of W and n = /t, + •• + n g . 
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It remains to apportion the error rate over the numerous confidence state¬ 
ments. Relation (5-28) still applies. There are p variables and g(g - l)/2 pairwise 
differences, so each two-sample f-interval will employ the critical value t n - g (a/2m ), 
where 

m = pg{g - l)/2 (6-46) 

is the number of simultaneous confidence statements. 

Result 6,5. Let n = n k ~ For the model in (6-38), with confidence at least 

*=l 

(1 - a), 

Tki ~ Ui belongs to x ki - x ei ± t n - g 

for all components i = 1,..., p and all differences £ < k = 1,..., g. Here w u is the 
ith diagonal element of W. 

We shall illustrate the construction of simultaneous interval estimates for the 
pairwise differences in treatment means using the nursing-home data introduced in 
Example 6.10. 



Example 6.11 (Simultaneous intervals for treatment differences—nursing homes) 

We saw in Example 6.10 that average costs for nursing homes differ, depending on 
the type of ownership. We can use Result 6.5 to estimate the magnitudes of the dif¬ 
ferences. A comparison of the variable X 3 , costs of plant operation and maintenance 
labor, between privately owned nursing homes and government-owned nursing 
homes can be made by estimating r 13 — t 33 . Using (6-39) and the information in 
Example 6.10, we have 



t 3 = (x 3 - x) 



'182.962 

= 4.408 8.200 

1.695 .633 1.484 

9.581 2.428 .394 6.538 


Consequently, 


r 13 - t 33 = -.020 - .023 = -.043 


and n = 271 + 138 + 107 = 516, so that 
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. o f or 950 / simultaneous confidence statements we require 

sim “ , “ n “ us 

dence statement is 

T 13 -T 33 belongsto^ . x . 

= -.043 ± 2.87(.00614) 

= -.043 ± -018, or (-.061, -.025) 

We conclude that jhe average0^^061 bday^h^for privately 

Ti3 - T 23 belongs to the interval (-.058, -.026) 

and T2J _ Tj3 belongs to the interval (-.021, .019) 

Thus, a difference in this cost exists between private and nonprofit nursing homes, 
but no difference is observed between nonprofit and government nursmg homes. . 


6.6 Testing for Equality of Covariance Matrices 

Oneoftheassumptionsmade when 

tors is that the we discuss discrimina- 

sampies to form a 

tion and class.ficabon Before pool ng ^ k can * worthwh ile to 

SKIS the population 

test for equal covariance matrices is Box s M-test ([ 8 ], (9J). 

With g populations, the null hypothesis is 

H 0 :X 1 =X 2 = ---=^ = S (6-47) 

thtTpreaimed*conimon covari ance alternative’hypothesis is that at least 

-JSSES5£S£S3£-.a———* <“ - 

ing (6-47) is given by (see (1]) 



Here n f is the sample size for the €th group, S f is the 6th group sample covariance 
matrix and S^m is the pooled sample covariance matrix given by 

- ._ L _f (fll -i)s 1 + («,-i)s 2 + - + K-i)s«} < 649) 

Spooled t v 1 

2 ><-1) 

i 


T ]3 ~ ?33 ± <513( 00208) 


J_ + 1 ] 

"! ny) n - g 
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Box’s test is based on his \ 2 approximation to the sampling distribution of -2 In A 
(see Result 5.2). Setting -2 In A = M (Box’s M statistic) gives 

M = [2(n, - 1)W | Spooled I - 2IK - l)ln|S«|] (6-50) 

If the null hypothesis is true, the individual sample covariance matrices are not 
expected to differ too much and, consequently, do not differ too much from the 
pooled covariance matrix. In this case, the ratio of the determinants in (6-48) will all 
be close to 1, A will be near 1 and Box’s M statistic will be small. If the null hypoth¬ 
esis is false, the sample covariance matrices can differ more and the differences in 
their determinants will be more pronounced. In this case A will be small and M will 
be relatively large. To illustrate, note that the determinant of the pooled covariance 
matrix, | S poo)ed |, will lie somewhere near the “middle” of the determinants | S € |’s of 
the individual group covariance matrices. As the latter quantities become more 
disparate, the product of the ratios in (6-44) will get closer toO. In fact, as the | |’s 

increase in spread, | |/| S pooJed | reduces the product proportionally more than 

I S( g ) |/| 8 ^^ | increases it, where | S<j) | and | S^) | are the minimum and maximum 
determinant values, respectively. 


Box’s Test for Equality of Covariance Matrices 


Y 1 _ 1 f 2p 2 + 3p - 1 

€(n e - 1 ) - 1 ) - P + “ 1 )' 

7 

where p is the number of variables and g is the number of groups. Then 


C = (1 - u)M = (1 - u)||2(n,-l)Jln | S^l - 2[(#i,-l)ln |S f |]J(6-52) 
has an approximate \ 2 distribution with 

v = s\p{p + 1) - |p(p + 1) = jp(p + 1)(£ - 1) (6-53) 

degrees of freedom. At significance level a, reject li 0 if C > A^(p+i)(^-iy 2 ( Q ')- 


Box’s x 2 approximation works well if each n e exceeds 20 and if p and g do not 
exceed 5. In situations where these conditions do not hold. Box ([7], [ 8 ]) has provided 
a more precise F approximation to the sampling distribution of M . 


Example 6.12 (Testing equality of covariance matrices — nursing homes) We intro¬ 
duced the Wisconsin nursing home data in Example 6.10. In that example the 
sample covariance matrices for p - 4 cost variables associated with g = 3 groups 
of nursing homes are displayed. Assuming multivariate normal data, we test the 
hypothesis H 0 : — X 2 = 2^3 = 2. 


312 Chapter 6 Comparisons of Several Multivariate Means 


Using the information in ExampJe 6.10, we have n Y = 271, n 2 = I 3 g 
m = 107and |S,| = 2.783 X 10' 8 ,|S 2 | = 89-539 X 10' 8 ,|S 3 | = 14.579 X 1CT 8 ,and 
I Spooled I = 17.398 X 10’ 8 Taking the natural logarithms of the determinants gives 
In | S, | = -17.397, In | Sj | = -13.926, In | S 3 | = -15.741 and In | S poo , ed | = -15.564. 
We calculate 

= [_L , _L . JL _ 1 _If 2(4 2 ) + 3(4) -1 1 = 

“ [270 137 106 270 + 137 + 106 JL 6(4 + 1)(3 - 1 ) J ' 

M = [270 +137 +106](—15.564) - [270(-17.397) +137(-13.926) + 106(-15.741)] 
= 289.3 

and C = (1 - .0133)289.3 = 285.5. Referring C to a * 2 table with v = 4(4 + 1 )(3 - l)/2 
= 20 degrees of freedom, it is clear that H 0 is rejected at any reasonable level of sig¬ 
nificance. We conclude that the covariance matrices of the cost variables associated 
with the three populations of nursing homes are not the same. ■ 

Box’s M- test is routinely calculated in many statistical computer packages that 
do MANOVA and other procedures requiring equal covariance matrices. It is 
known that the Af-test is sensitive to some forms of non-normality. More broadly, in 
the presence of non-normality, normal theory tests on covariances are influenced by 
the kurtosis of the parent populations (see [16]). However, with reasonably large 
samples, the MANOVA tests of means or treatment effects are rather robust to 
nonnormality. Thus the Af-test may reject H 0 in some non-normal cases where it is 
not damaging to the MANOVA tests. Moreover, with equal sample sizes, some 
differences in covariance matrices have little effect on the MANOVA tests. To 
summarize, we may decide to continue with the usual MANOVA tests even though 
the A/-test leads to rejection of H 0 . 

6.7 Two-Way Multivariate Analysis of Variance 

Following our approach to the one-way MANOVA, we shall briefly review the 
analysis for a univariate two-way fixed-effects model and then simply generalize to 
the multivariate case by analogy. 

Univariate Two-Way Fixed-Effects Model with Interaction 

We assume that measurements are recorded at various levels of two factors. In some 
cases, these experimental conditions represent levels of a single treatment arranged 
within several blocks. The particular experimental design employed will not concern 
us in this book. (See [10] and [17] for discussions of experimental design.) We shall, 
however, assume that observations at different combinations of experimental condi¬ 
tions are independent of one another. 

Let the two sets of experimental conditions be the levels of, for instance, factor 
1 and factor 2 , respectively . 4 Suppose there are g levels of factor 1 and b levels of fac¬ 
tor 2 ,and that n independent observations can be observed at each of the gb combi¬ 
ne use of the term “factor” to indicate an experimental condition is convenient. The factors dis¬ 
cussed here should not be confused with the unobservable factors considered in Chapter 9 in the context 
of factor analysis. 


Expected response Expected response 
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nations of levels. Denoting the rth observation at level £ of factor 1 and level k of 
factor 2 by Xtkr, we specify the univariate two-way model as 

Xtkr = + Pk + Jtk + e €kr 

i:\izi 

r = 1 , 2 ,. .. ,n 

8 b g b 

where 2/ t ( = 2i /3 k = 2, y ek = 2, y (k = 0 and the e tkr are independent 

f=l k =1 £-1 *=1 

N( 0 , a 2 ) random variables. Here 11 represents an overall level, r € represents the 
fixed effect of factor 1 , p k represents the fixed effect of factor 2 , and y €k i nter ‘ 
action between factor 1 and factor 2. The expected response at the £th level of factor 
1 and the fcth level of factor 2 is thus 


E(X ekr ) 


response 


overall 

level 


effect of 
factor 1 


effect of 
factor 2 


factor 1 -factor 2 
interaction 


* = 1 , 2 ,... ,g, k = 1 , 2 ,... ,6 


The presence of interaction, y (k , implies that the factor effects are not additive 
and complicates the interpretation of the results. Figures 6.3(a) and (b) show 


Level 1 of factor 1 
Level 3 of factor 1 
Level 2 of factor 1 


Level of factor 2 



, Level 3 of factor 1 


Level 1 of factor 1 


Level 2 of factor 1 


Level of factor 2 


Figure 6.3 Curves for expected 
responses (a) with interaction and 
(b) without interaction. 
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expected responses as a function of the factor levels with and without interaction, 
respectively. The absense of interaction means y (k = 0 for all f and k. 

In a manner analogous to (6-55), each observation can be decomposed as 

x Ur = x + ( Xf. - x) + (*.* - x) + (x tk - X( - x.k + x) + (xekr - x n ) (6-56)- 

where x is the overall average, xe- is the average for the fth level of factor 1 , x. k is 
the average for the kth level of factor 2 , and x €k is the average for the f’th level of 
factor 1 and the kth level of factor 2, Squaring and summing the deviations 

(x €k r - *) g> ves 

£ X 2 - xf = 2 + 2 sn{x- k - xf 

C=1 *=1 r=l f=l *-l 

8 b _ 2 
+ 22 n ( x €k - x e • ■“ X'k. + x ) 

C= 1 k = 1 

+ J 2 2 (Xftr - X P *) 2 (6-57) 

0=1 *=l r=l 


SScor = SS facl + SS fac2 + SS int + SS res 

The corresponding degrees of freedom associated with the sums of squares in the J 
breakup in (6-57) are 

gbn - 1 = {g ~ 1) + (b - 1) + (g - l)(b - 1) + gb(n - 1) (6-58) 

The ANOVA table takes the following form: 


ANOVA Table for Comparing Effects of Two Factors and Their Interaction 


Source 
of variation 

Sum of squares (SS) 

Degrees of 
freedom (d.f.) 

Factor 1 

ss facl = 2 - x) 2 

g ~ 1 

Factor 2 

» 2 

SS fac2 = 2 gn(x. k - x) 

k— 1 

b - 1 

Interaction 

g b ^ 

SS in , = 22 n (*e k - Xf. - x. k + x) 

€*1 A: = l 

( g - 1 )(b - 1 ) 

Residual (Error) 

SS res =^22 (x«r - Xf *) 2 

^1 k- 1 f = l 

gb(n - 1 ) 

Total (corrected) 

SS^s = ^ X E (X«r - X) 2 

k~\ r=l 

gbn - 1 
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The F-ratios of the mean squares, SS facl /(g - 1), SS fac2 /(b '1), and 
SSi nt /(g ~ l)(/> — 1 ) to the mean square, S S r&s /(gb(n - 1 )) can be used to test for 
the effects of factor 1, factor 2, and factor 1-factor 2 interaction, respectively. (See 
[ 11 ] for a discussion of univariate two-way analysis of variance.) 


Multivariate Two-Way Fixed-Effects Model with Interaction 

Proceeding by analogy, we specify the two-way fixed-effects model for a vector 
response consisting of p components [see (6-54)] 

^ekr - fx + + Pk + Ytk + *ek r 

f=1,2 ’ -,g (6-59) 

k = 1 , 2 ,..., 

r = 1,2,...,* 

8 b 8 b 

where X T c = 2 Pk = 2 Tf/t = S Te* = 0. The vectors are all of order p X 1, 

f =] * = 1 €=1 

and the e^,. are independent N p ( 0, X) random vectors. Thus, the responses consist 
of p measurements replicated n times at each of the possible combinations of levels 
of factors 1 and 2 . 

Following (6-56), we can decompose the observation vectors x ikr as 

*f*r = x + (*f- x) + (*•* ~ x) + (x tk -x { . ~x. k +x) + ( x (kr - x € *) (6-60) 

where x is the overall average of the observation vectors, x^. is the average of the 
observation vectors at the fth level of factor 1 , X.* is the average of the observation 
vectors at the kth level of factor 2, and x n is the average of the observation vectors 
at the fth level of factor 1 and the kth level of factor 2 . 

Straightforward generalizations of (6-57) and (6-58) give the breakups of the 
sum of squares and cross products and degrees of freedom: 

I 2 i (*ftr ~ *)(*C*r - *)' = 2 ~ *)(*<• * X)' 

£=] * = l r=1 €=1 

b 

+ 2 g"(x-t - x)(x.* ~ x)' 

Jk=l 


+ 2 S «(*«- *<• - x.* + x)(5f*~ Xf.~ X. k + x)' 

<=1 *=1 

+ ^ 2 2 (x«r - *€*)(*«. - Xf*)' (6-61) 

1 A:=l r=l 

- 1 = (f - 1) + (b - 1) + (g - 1) (b - 1) + gb(n - 1) (6-62) 

Again, the generalization from the univariate to the multivariate analysis consists 
simply of replacing a scalar such as (3^. - x) 2 with the corresponding matrix 
(Xf. - x)(Xf. - x)'. 
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The MANOVA table is the following: 
MANQVA Table for Comparing Factors and Their Interaction 


Degrees of 


Source of 
variation 

Matrix of sum of squares 
and cross products (SSP) 

freedom 

(df.) 

Factor 1 

SSP fac , = i bn(x e . - x) (if ~ *)’ 

8~ 1 

Factor 2 

b 

ssp fac2 = S £«(*•* - *) (*•* _ *)' 

k=l 

b - 1 

Interaction 

Residual 

(Error) 

SSP in , = £ 2 «(*<* ~ ** ~ *•* + *)(*« “ ~ *'* + 

f = l Jt=l 

SSP re s = ^ 2 2 (*«<■ ~ “ *«)' 

k = 1 r~ 1 

(g-l)( 6 - 

gb(n - 1) 

Total 

(corrected) 

ssp cor = i; 2 2 - *)' 

f=l k = \ r=1 

gbn - 1 




A test (the likelihood ratio test) 5 of 

Ho- 711 = ri 2 = • ■ • = Vgb = 0 (no interaction effects) ( 6 -^ 

versus M 

Hi : At least one y n * 0 ^ 

is conducted by rejecting H 0 for small values of the ratio 

a * = | SSPres I (6-64) 

ISSP^-f SSP re5 | 

For large samples, Wilks’ lambda. A*, can be referred to a 
Using Bartlett’s multiplier (see [6]) to improve the chi-square appro . , 

reject H 0 : Yu = 712 = ’ ’ ’ = Ygb = 0 at the a ,evcl 


where A* is given by (6-64) and x( g -i)(b-i)p( a ) 1S 1116 u PP er ( 100a ) th P ercentile *g 

chi-square distribution with (g - l)(b - l)pd.f. factoregl 

Ordinarily, the test for interaction is carried out before the tests for m^adojj 
fects. If interaction effects exist, the factor effects do not mulj 

From a practical standpomt, it is not advisable to proceed wi or each var jal|i 
variate tests. Instead ,p univariate two-way analyses of vanance ( but|cS 

are often conducted to see whether the interaction appears in some respons J| 

5 The likelihood test procedures require that p s gb(n - 1), so that SSP ra will be po 
(with probability 1). 
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others. Those responses without interaction may be interpreted in terms of additive 
factor 1 and 2 effects, provided that the latter effects exist. In any event, interaction 
plots similar to Figure 6.3, but with treatment sample means replacing expected values, 
best clarify the relative magnitudes of the main and interaction effects. 

In the multivariate model, we test for factor 1 and factor 2 main effects as 
follows. First, consider the hypotheses H 0 : Tj = t 2 = * * * = r g = 0 and H Y : at least 
one T( * 0. These hypotheses specify no factor 1 effects and some factor 1 effects, 
respectively. Let 


|SSP facl + SSP res | 

so that small values of A* are consistent with H x . Using Bartlett’s correction, the 
likelihood ratio test is as follows: 

Reject H 0 :ti = t 2 = * * • = r g = 0 (no factor 1 effects) at level a if 

- gb(n - 1) - -- -- ln A * > *(s-i)p(“) (6-67) 

where A* is given by ( 6 - 66 ) and x}g-i) P ( a ) is the u PP e r (100a)th percentile of a 
chi-square distribution with ( g - 1 )p d.f. 

In a similar manner, factor 2 effects are tested by considering H 0 ‘ Pi = 
p 2 ~ • * ’ = Pb “ ® and Hi', at least one p k * 0 . Small values of 

A* = 7 --- ( 6 - 68 ) 

I SSP fac2 + SSP ri;s | 

are consistent with Hi . Once again, for large samples and using Bartlett’s correction: 
Reject H 0 : p x = p 2 = * * ■ = fit, = 0 (no factor 2 effects) at level a if 

- gb{n - 1) - - + 1 ~ ln A * > X(b-i)p(<*) ( 6 ‘ 69 ) 

where A* is given by ( 6 - 68 ) and X(b-i) p (ot) is the upper (100a)th percentile of a 
chi-square distribution with (b — 1 )/? degrees of freedom. 

Simultaneous confidence intervals for contrasts in the model parameters 
can provide insights into the nature of the factor effects. Results comparable to 
Result 6.5 are available for the two-way model. When interaction effects are 
negligible, we may concentrate on contrasts in the factor 1 and factor 2 main 
effects. Hie Bonferroni approach applies to the components of the differences 
?€ ~ Tm of factor 1 effects and the components of p k - p q of the factor 2 
effects, respectively. 

The 100(1 — a)% simultaneous confidence intervals for tu — r mi are 


belongs to (x^ - x m .i) ± t v 


pg(g - i)/ 


iTTn (6 ‘ 70) 


where v - gb(n - 1 ), E ti is the ith diagonal element of E = SSP res , and - x m .i 
is the ith component of x t . - x m . . 
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Similarly, the 100(1 - a) percent simultaneous confidence intervals for p ki - f} q 


Pki-Pgi belongs to (*•*, - •*•,/) ± ‘>{ pb ( b -i))\ v gn ((W1) 

where v and are as just defined and x. ki - I. ? , is the fth component of x.* - x. q . 

Comment We have considered the multivariate two-way model with replica¬ 
tions. That is, the model allows for n replications of the responses at each combina¬ 
tion of factor levels. This enables us to examine the “interaction of the factors. If 
only one observation vector is available at each combination of factor levels, the 
two-way model does not allow for the possibility of a general interaction term y tk . 
The corresponding MANOVA table includes only factor 1, factor 2, and residual 
sources of variation as components of the total variation. (See Exercise 6.13.) 

Example 6.13 (A two-way multivariate analysis of variance of plastic film data) The 

optimum conditions for extruding plastic film have been examined using a tech¬ 
nique called Evolutionary Operation. (See [9].) In the course of the study that was 
done, three responses— X x = tear resistance, X 2 - gloss, and X 2 - opacity—were 
measured at two levels of the factors, rate of extrusion and amount of an additive. 
The measurements were repeated n = 5 times at each combination of the factor 
levels. The data are displayed in Table 6.4. 

Table 6 .4 Plastic Film Data _ 

Xi = tear resistance, x 2 = gloss, and x 3 = opacity __ 

1 | Factor 2: Amount of additive 


Low (-10)% 


Factor 1: Change 
in rate of extrusion 


Low (1.0%) 

High (1.5%) 


f2 

£3 

£i 

£2 

£3 

[6.5 

9.5 

4.4 

6.9 

9.1 

5.7] 

[6.2 

9.9 

6.4 

7.2 

10.0 

2.0] 

[5.8 

9.6 

3.0 

6.9 

9.9 

3.9] 

[6.5 

9.6 

4.1 

6.1 

9.5 

1.9] 

[6.5 

9.2 

0.8 

[6.3 

9.4 

5.7] 

£1 

h 

£3 

£i 

£2 

£3 

[6.7 

9.1 

2.8 

7.1 

9.2 

8.4] 

[6.6 

9.3 

4.1 

7.0 

8.8 

5.2] 

[7.2 

8.3 

3.8 

7.2 

9.7 

6.9] 

[71 

8.4 

1.6 

7.5 

10.1 

2.7 

[6.8 

8.5 

3.4 

7.6 

9.2 

1.9] 


High (10%) 


The matrices of the appropriate sum of squares and cross products were calcu¬ 
lated (see the SAS statistical software output in Panel 6.1 ), leading to the following 
MANOVA table: 

‘Additional SAS programs for MANOVA and other procedures discussed in this chapter are 
available in [13]. 
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Source of variation 


Factor 1: 


Factor 2: 


change in rate 
of extrusion 

amount of 
additive 


Interaction 


Residual 


Total (corrected) 


-.7855 

5.0855 


.8555 

-.7395 

.4205 

1.9305 

1.7325 

4.9005 

.0445 

1.4685 

3.9605 

-3.0700 

-.5520 

64.9240 

-.2395 

1.9095 

74.2055 


PANEL 6.1 SAS ANALYSIS FOR EXAMPLE 6.13 USING PROC GLM 

title 'MANOVA'; 
data film; 
infile T6-4.dat'; 
input xi x2 x3 factor 1 factor2; 

proc glm data = film; PROGRAM COMMANDS 

class factorl factor2; 

model xi x2 x3 = factorl factor2 factorl *factor2 /ss3; 
manova h = factorl factor2 factorl*factor2 /printe; 
means factorl factor2; 


| Dependent Variable: XI 


General Linear Models Procedure 
Class Level Information 

Class Levels Values 

FACTOR 1 2 0 1 

FACTOR2 2 0 1 

Number of observations in data set = 20 


Source 

DF Sum of Squares 

Mean Square 

F Value 

Pr > F 

Model 

3 

2.50150000 

0.83383333 

7.56 

0.0023 

Error 

16 

1.76400000 

0.11025000 



Corrected Total 

19 

4.26550000 





R-Square 

C.V. 

Root MSE 


XI Mean 


0.586449 

4.893724 

0.332039 


6.78500000 

Source 

DF 

Type III SS 

Mean Square 

F Value 

Pr > F 


1 

1 

1.74050000 

0.76050000 

1.74050000 

0.76050000 

15.79 

6.90 

0.0011 
0.0183 

_, 

1 

0.00050000 

0.00050000 

0.00 

0.9471 


(continues on next page) 
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PANEL 6.1 (continued) 


Dependent Variable: X2 

Source 

Model 

Error 

Corrected Total 


[FACTOR! 

FACTOR2 _ 

FACTORI »FACTOR2 

Dependent Variable: X3 

Source 

Model 

Error 

Corrected Total 


FACTOR 

FACTOR 2 _ 

FACTORI *FACTOR2 


DF Sum of Squares Mean Square F Value 
3 2.45750000 0.81916667 4.99 

16 2.62800000 0.16425000 

19 5.08550000 


R-Square 

0.483237 


C.V. 

4350807 


Root MSE 
0.405278 


Type III SS Mean Square 


1.30050000 I 
0-61250000 | 
0.54450000 


1.30050000 

0.61250000 

0.54450000 


DF Sum of Squares Mean Square F Value 
3 9.28150000 3.09383333 0.76 

16 64.92400000 4.05775000 

19 74.20SSOOOO 


R-Square 

0.125078 


C.V. 

51.19151 


Root MSE 
2.014386 


Type HI SS Mean Square F Value 


0.42050000 

4.90050000 

3.96050000 

E a Error 5S&CP Matrix 


0.420S000O 

4.90050000 

3.960S00Q0 


XI 

X2 

X3 

1.764 

0.02 

-3.07 

0.02 

2.628 

-0.552 

-3.07 

-0.552 

64.924 


M a nova Test Criteria and Exact F Statistics for 
the Hypothesis of no Overall FACTORI Effect 


H = Type III S5&CP Matrix for FACTORI 
S=1 M = 0.5 


E = Error 5S&CP Matrix 
N = 6 


Statistic 

Wilks' Lambda . 

Pillai's Trace 

Hotelling-Lawley Trace 
Roy's Greatest Root 


"Value 

F 

NumDF 

Den DF 

Pr>f;5] 


7.5543 

3 

14 

oxw30:-. 

0.61814162 

7.5543 

3 

14 

0.0030 

1.61877188 

7.5543 

3 

14 

0.0030 

1.61877188 

7.5543 

3 

14 

0.0030 




X2 Mean 
9.31S00000 


X3 Mean 
3.93500000 


{mntinups on next PdQ^ s 
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PANEL 6.1 (continued) 


Manova Test Criteria and Exact F Statistics for 
the [Hypothesis of no Overall FACTOR2 Effect 


H = Type III SS&CP Matrix for FACTOR2 
S = 1 M = 0.5 


pillai's Trace 
Hotelling-Lawley Trace 
Roy's Greatest Root 


E = Error SS&CP Matrix 
N =6 


Statistic l 

Value 

F 

Num DF 

Den DF 

Pr > F 

Wilks' lambda 

0.52303490 

4.2556 

3 

14 

0.0247 


0.47696510 

0.91191832 

0.91191832 


Manova Test Criteria and Exact F Statistics for 

the Hypothesis of no Overall FACTORI*FACTOR2 Effect 

H = Type III SS&CP Matrix for FACTOR 1*FACTOR2 E = Error SS&CP Matrix 
S = 1 M = 0.5 N = 6 


Statistic 

Value 

F 

Num DF 

Den DF 

Pr > F 

Wilks' Lambda 

0.77710576 

1.3385 

3 

14 

0.3018 


Pillai's Trace 
Hotelling-Lawley Trace 
Roy's Greatest Root 

Level of 
FACTORI 
0 
1 


0.22289424 

0.28682614 

0.28682614 


Level of 
FACTOR2 
0 
1 


N 

Mean 

SD 

Mean 

SD 

10 

6.49000000 

0.42018514 

9.57000000 

0.29832868 

10 

7.08000000 

0.32249031 

9.06000000 

0.57580861 

Level of 



X3- 

_ 

FACTORI 

N 

Mean 

SD 


0 

10 

3.79000000 

1.85379491 


1 

10 

4.08000000 

2.18214981 


N 

Mean 

-XI- 

SD 

Mean 

- X2- 

SD 

10 

6.59000000 

0.40674863 

9.14000000 

0.56015871 

10 

6.98000000 

0.47328638 

9.49000000 

0.42804465 

Level of 



X3- 


FACTOR2 

N 

Mean 

5D 


0 

10 

3.44000000 

1.55077042 


1 

10 

4.43000000 

2.30123155 



To test for interaction, we compute 


275.7098 

354.7906 


= .7771 
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For (g - 1 )(b - 1) = 1, 


F = 


1 - AM (gb{n - 1 ) - p + l )/2 
A* 7 Cite - 1)(*- 1) -p| + l)/2 


has an exact F-distribution with = |(g - l)(b - 1 ) - p\ + 1 and ^ 
gb(n - 1) - p + 1 d.f. (See [1].) For our example. 




1 - .7771 \ (2(2)(4) - 3 + l)/2 


.7771 ){ |1(1)-3| +l)/2 
v \ ~ (11(1) 3 1 H- 1) = 3 

v i ~ (2(2) (4) — 3 + 1) = 14 


= 134 


the? 


and F 3 j 14 (.05) = 3.34. Since F = 1.34 < F 314 (.05) = 334, we do not reject 
hypothesis H 0 : yu ~ y i 2 ~ 721 ~ 722 = 0 (no interaction effects). 

Note that the approximate chi-square statistic for this test is ~[2(2)(4)^ 
(3 + 1 - l(l))/2] ln(.7771) = 3.66, from (6-65). Since ^(.05) = 7.81, we 
reach the same conclusion as provided by the exact F-test. 

To test for factor 1 and factor 2 effects (see page 317), we calculate 


: WOUJd, .” 


and 


AT = 


Aj = 


|SSP re 


SSP facl + SSP re 
|SSP res | 


275.7098 
' 722.0212 


.3819 


275.7098 


= .5230 


|SSP fac2 + SSP res | 527.1347 

For both g — 1 = 1 and b -1 = 1, 

= ( . 1 ~ AlWftfr ~ 1) ~ P + l)/2 

a : ) (\(g-\)-p\ + m 

1 - aA ( gb(n - 1) - p + l)/2 

A * 2 J (K* - 1) - P\ + l)/2 


and 


Ft = 


R = 


have F-distributions with degrees of freedom ^i = |(g-l) - Fl+l» v 2 = - -J 
gb (n - 1 ) - p + 1 and jq = | (6 - 1 ) - p\ + l,vz = gb(n - 1) - p + l,respec- 
tively (See [1].) In our case, 


and 


Fl = 1 

^1 - .3819 \ 
V .3819 ) 

(16-3 +l)/2 
(|l-3| + l)/2 

F 2 = | 

(1 - .5230N 

(16 - 3 + l)/2 _ ^ 

( .5230 ) 

(|1 -3( +1)/2 

= 11 ~ 

3| + 1 =3 

v 2 ~ (16 - 3 + 1) = 


From before, F 3 ,i 4 (-05) = 3.34. We have Fj = 7.55 > F 3>14 (.05) = 3.34, and 
therefore, we reject H 0 : r\ = r 2 = 0 (no factor 1 effects) at the 5% level. Similarly, 
F 2 = 4.26 > F 3 14 (.05) = 3.34, and we reject Hq\ = p 2 ~ 0 (no factor 2 effects) 
at the 5% level. We conclude that both the change in rate of extrusion and the amount 
of additive affect the responses, and they do so in an additive manner. 

The nature of the effects of factors 1 and 2 on the responses is explored in Exer¬ 
cise 6.15. In that exercise, simultaneous confidence intervals for contrasts in the 
components of re and p k are considered. ■ 


Profile Analysis 

Profile analysis pertains to situations in which a battery of p treatments (tests, ques¬ 
tions, and so forth) are administered to two or more groups of subjects. All responses 
must be expressed in similar units. Further, it is assumed that the responses for the 
different groups are independent of one another. Ordinarily, we might pose the 
question, are the population mean vectors the same? In profile analysis, the question 
of equality of mean vectors is divided into several specific possibilities. 

Consider the population means fx\ = [mi 1 > Mi 2 > Mi 3 , Mi 4 ] representing the average 
responses to four treatments for the fust group. A plot of these means, connected by 
straight lines, is shown in Figure 6.4.This broken-line graph is the profile for population 1. 

Profiles can be constructed for each population (group). We shall concentrate 
on two groups. Let Mi = [mu, Mi2> • • • > Mip] and M 2 * [M 2 n M 22 >- ••, M 2 p] be the 
mean responses to p treatments for populations 1 and 2, respectively. The hypothesis 
H 0 : Mi = /t 2 implies that the treatments have the same (average) effect on the two 
populations. In terms of the population profiles, we can formulate the question of 
equality in a stepwise fashion. 

1. Are the profiles parallel? 

Equivalently: Is 7/ 0 1 :Mi/ ~ Mii-i = M 2 M 2 i-i» * = 2,3,...,/?, acceptable? 

2. Assuming that the profiles are parallel, are the profiles coincident? 7 

Equivalently: Is // 02 : mi/ = M 2 /> * = 1,2,...,/?, acceptable? 


Mean 

response 



7 The question, “Assuming that the profiles are parallel, are the profiles linear?” is considered in 
Exercise 6.12. The null hypothesis of parallel linear profiles can be written Hq. (fiy, + f* 2 i) 

- (M 1/-1 + M 2 i-i) - + /A 2 i-i) ~ (Pu-2 + ^ 2 .- 2 )- 1 =3 . p. Although this hypothesis may be 

of interest in a particular situation, in practice the question of whether two parallel profiles are the same 
(coincident), whatever their nature, is usually of greater interest. 







324 Chapter 6 Comparisons of Several Multivariate Means 


3. Assuming that the profiles are coincident, are the profiles level? That is, are all 
the means equal to the same constant? 

Equivalently: Is H Q3 : fi n = /x 12 = ■ * * = fi lp = = *'* = acceptable? 

The null hypothesis in stage 1 can be written 
H 01 : C/t] = C/t 2 

where C is the contrast matrix 

~-l 1 0 0 0 0 

= 0—110 00 
((p-i)xp) : : : * 

0 0 0 0 * • -1 1 

For independent samples of sizes n Y and n 2 from the two populations, the null 
hypothesis can be tested by constructing the transformed observations 

Cx ly , j = 

and 

Cx 2y , j ~ l,2,...,n 2 

These have sample mean vectors Ci^ and Cx 2 , respectively, and pooled covariance 
matrix CS pooled C'. 

Since the two sets of transformed observations have 7V p _ 1 (C/Ai, CXC') and 
N p ^(Cfi 2 , CXC') distributions, respectively, an application of Result 6.2 provides a 
test for parallel profiles. 



Test for Parallel Profiles for Two Normal Populations 


Reject Hoi: C/*i = C/t 2 (parallel profiles) at level a if 

T 2 = (Xj - x 2 )'C' f + ^W 00led C' C(x, - x 2 ) > c 2 (6-73) 


A - ("i + ” 2 ~ 2)(p ~ ^ r /_} 

C ~ m + n 2 - P Fp-l.n 1+ n 2 -p(<*) 


When the profiles are parallel, the first is either above the second (/ii, > >cl 2i , 
for all /), or vice versa. Under this condition, the profiles will be coincident only if 
the total heights fi n + /i 12 + • • • + fii p = l'/tti and M 21 + M 22 + • * ■ + fx 2p = l '/*2 
are equal. Therefore, the null hypothesis at stage 2 can be written in the equivalent 
form 

tf()2 : l'/*l = l'M2 

We can then test H 02 with the usual two-sample /-statistic based on the univariate 
observations i'xj p j = 1,2and l'x 2y , j = 1,2,..., n 2 . 
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Test for Coincident Profiles, Given That Profiles Are Parallel 

For two normal populations, reject Hq 2 : 1'fXi = l'/* 2 (profiles coincident) at 
level a if 

t 2 = i'(xi - x 2 ) [ —+ —Jr 
LV"i ”2/ 

m - g 2> \ 

+ ^) 1Spoo ' ed1 / 




For coincident profiles, x u , x 12 ,..., x lnj and x 21 , x 22 ,..., x 2 „ 2 are all observa¬ 
tions from the same normal population? The next step is to see whether all variables 
have the same mean, so that the common profile is level. 

When H 01 and H 02 are tenable, the common mean vector fx is estimated, using 
all n Y -I- n 2 observations, by 


+ ”2 \/=i 


+ 2 *2/ = 
M / 


(«l + " 2 ) 1 («i + " 2 ) 


If the common profile is level, then /q = fi 2 = • • * = ix p , and the null hypothesis at 
stage 3 can be written as 


Hq 3 . C/t — 0 


where C is given by (6-72). Consequently, we have the following test. 


Test for Level Profiles, Given That Profiles Are Coincident 

For two normal populations: Reject H 03 : C/t = 0 (profiles level) at level a if 

(/j, + n 2 )x'C'[CSC']“ 1 Cx > c 2 (6-75) 

where S is the sample covariance matrix based on all n x + n 2 observations and 


c 


2 


(/j, + n 2 - 1 )(p - 1) 

V ni 4J;%V W2 w«) 


Example 6.14 (A profile analysis of love and marriage data) As part of a larger study 
of love and marriage, E. Hatfield, a sociologist, surveyed adults with respect to their 
marriage “contributions” and “outcomes” and their levels of “passionate” and 
“companionate” love. Recently married males and females were asked to respond 
to the following questions, using the 8-point scale in the figure below. 



1 2 3 4 5 6 7 8 
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1 . All things considered, how would you describe your contributions to the 
marriage? 

2. All things considered, how would you describe your outcomes from the 
marriage? 

Subjects were also asked to respond to the following questions, using the 
5 -point scale shown. 

3. What is the level of passionate love that you feel for your partner? 

4. What is the level of companionate Jove that you feel for your partner? 

None Very A great Tremendous 

at all little Some deal amount 


x r = an 8 -point scale response to Question 1 
x 2 = an 8 -point scale response to Question 2 
*3 = a 5-point scale response to Question 3 
jc 4 = a 5-point scale response to Question 4 

and the two populations be defined as 

Population 1 = married men 
Population 2 = married women 

The population means are the average responses to the p — 4 questions for the 
populations of males and females. Assuming a common covariance matrix X, it is of 
interest to see whether the profiles of males and females are the same. 

A sample of n { = 30 males and n 2 = 30 females gave the sample mean vectors 


6.833~ 


‘6.633’” 

7.033 


7.000 

3.967 

> x 2 = 

4.000 

4.700_ 


_4.533_ 

(males) 


(females) 


and pooled covariance matrix 


.606 

.262 

.066 

.161 

.262 

.637 

.173 

.143 

.066 

.173 

.810 

.029 

.161 

.143 

.029 

.306 


The sample mean vectors are plotted as sample profiles in Figure 6.5 on page 327. 

Since the sample sizes are reasonably large, we shall use the normal theory 
methodology, even though the data, which are integers, are clearly nonnormal. Tb 
test for parallelism (tf 0 i : 0*1 = Cfi 2 ), we compute 
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Sample mean 
response x ej 


Key: 

X-X Males 

O— — o Females 


Figure 6.5 Sample profiles 
for marriage-love responses. 


^-'S DOO i ef ]C — 


C(xi - x 2 ) = 


0 0 -1 lj 

.719 -.268 -.125 

-.268 1.101 -.751 
-.125 -.751 1.058 


-110 0 
0-1 10 
0 0-11 



-1 

0 

o" 


1 

-1 

0 

Spooled 

0 

1 

-1 


0 

0 

1^ 


.719 -.268 -.125 

T 2 = [-.167,-.066, .200] + i ) _1 -.268 1.101 -.751 

_--125 -.751 1.058 

= 15(.067) = 1.005 


1 T -.167 
-.066 
.200 


Moreover, with a= .05, c 2 = [(30+30-2)(4-l)/(30+30-4)]F 3>56 (.05) = 3 . 11 ( 2 . 8 ) 
— 8.7. Since T 2 = 1.005 < 8.7, we conclude that the hypothesis of parallel profiles 
for men and women is tenable. Given the plot in Figure 6.5, this finding is not 
surprising. 

Assuming that the profiles are parallel, we can test for coincident profiles. To 
test H 02 : l'ni = l'/* 2 (profiles coincident), we need 

Sum of elements in (xj - x 2 ) = l'(xj - x 2 ) = .367 
Sum of elements in S p00led = l'S pooIed l = 4.207 
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Using (6-74), we obtain 

367 

V& + £)4.027 

With a = .05, F 1j 58 (.G5) - 4.0, and T 2 = .501 < Fi > 58 (.05) = 4.0, we cannot reject 
the hypothesis that the profiles are coincident. That is, the responses of men and 
women to the four questions posed appear to be the same. 

We could now test for level profiles; however, it does not make sense to carry 
out this test for our example, since Questions 1 and 2 were measured on a scale of 
1-8, while Questions 3 and 4 were measured on a scale of 1-5. The incompatibility of 
these scales makes the test for level profiles meaningless and illustrates the need for 
similar measurements in order to carry out a complete profile analysis. a 

When the sample sizes are small, a profile analysis will depend on the normality 
assumption.This assumption can be checked, using methods discussed in Chapter 4, 
with the original observations or the contrast observations Cx € j. 

The analysis of profiles for several populations proceeds in much the same 
fashion as that for two populations. In fact, the general measures of comparison are 
analogous to those just discussed. (See [13], [18].) 


6.9 Repeated Measures Designs and Growth Curves 

As we said earlier, the term “repeated measures” refers to situations where the same 
characteristic is observed, at different times or locations, on the same subject. 

(a) The observations on a subject may correspond to different treatments as in 
Example 62 where the time between heartbeats was measured under the 2 X 2 
treatment combinations applied to each dog. The treatments need to be com¬ 
pared when the responses on the same subject are correlated. 

(b) A single treatment may be applied to each subject and a single characteristic 
observed over a period of time. For instance, we could measure the weight of a 
puppy at birth and then once a month. It is the curve traced by a typical dog that 
must be modeled. In this context, we refer to the curve as a growth curve. 

When some subjects receive one treatment and others another treatment, 
the growth curves for the treatments need to be compared. 

To illustrate the growth curve model introduced by Potthoff and Roy [21], we 
consider calcium measurements of the dominant ulna bone in older women. Besides 
an initial reading, Table 6.5 gives readings after one year, two years, and three years 
for the control group. Readings obtained by photon absorptiometry from the same 
subject are correlated but those from different subjects should be independent. The 
model assumes that the same covariance matrix X holds for each subject. Unlike 
univariate approaches, this model does not require the four measurements to have 
equal variances. A profile, constructed from the four sample means (jtj, x 2 , * 3 , *4), 
summarizes the growth which here is a loss of calcium over time. Can the growth 
pattern be adequately represented by a polynomial in time? 




Repeated Measures Designs and Growth Curves 329 



When the p measurements on all subjects are taken at times t u t 2> . -. 
Potthoff-Roy model for quadratic growth becomes 


J P , the 


£0 + 0i*i + 02*1 
£0 + 0i*2 + p 2 t 2 

0o + 0i t p + p 2 tl 


where the ith mean p t is the quadratic expression evaluated at f,. 

Usually groups need to be compared. Table 6.6 gives the calcium measurements 
for a second set of women, the treatment group, that received special help with diet 
and a regular exercise program. 

When a study involves several treatment groups, an extra subscript is needed as 

in the one-way MANOVA model. Let X n , X e2 .X« B be the n € vectors of 

measurements on the n € subjects in group €, for € - 1,,.., g. 

Assumptions. All of the X^j are independent and have the same covariance 
matrix X. Under the quadratic growth model, the mean vectors are 


Peo + 0ei*i + 0£2*i 1 h t\ 

0eo + Peih + 0 ^ 2*2 1 t 2 t\ 


E[X €} ] = 


.Peo + Pe\*p + Pei * 2 p. 


Pto 

Pn ~ B0f 
0 € 2 _ 
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where 



If a ^th-order polynomial is fit to the growth data, then 



(6-76) 


(6-77) 


Under the assumption of multivariate normality, the maximum likelihood 
estimators of the p e are 

fit = (B'Sj& w B)" 1 B'Sj£ lW X, for £=1,2 . g (6-78) 

where 


Spooled = ^-r-y(("! -l)Si + -"+ (n g - 1 ) 8 ,) = ^ 7 ^ 
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with N = ^ n e > i s the pooled estimator of the common covariance matrix S. The 

e=i 

estimated covariances of the maximum likelihood estimators are 


Co y(fi t ) = -(B'S^ oled B)' 
n e 


£ = 1 , 2 ,... 


where A: =JN - g)(N - g - 1 )/(N - g - p + q)(N - g - p + q + 1). 

Also, and p h are independent, for i * h, so their covariance is 0. 

We can formally test that a <?th-order polynomial is adequate. The model is fit 
without restrictions, the error sum of squares and cross products matrix is just the 
within groups W that has N - g degrees of freedom. Under a gth-order polynomi¬ 
al, the error sum of squares and cross products 

w, = 2 2 (Xf; - B Pe)(Xej - B p e Y (6-80) 

£=1 ; = 1 

has n g -g+p-q - 1 degrees of freedom. The likelihood ratio test of the null 
hypothesis that the <?-order polynomial is adequate can be based on Wilks 5 lambda 


Under the polynomial growth model, there are q + 1 terms instead of the p means 
for each of the groups. Thus there are (p ~ q - l)g fewer parameters. For large 
sample sizes, the null hypothesis that the polynomial is adequate is rejected if 

~(w - |(P - q + g)) |n A* > X(p-q- i)g(a) (6-82) 


Example 6.15 (Fitting a quadratic growth curve to calcium loss) Refer to the data in 
Tables 6.5 and 6.6. Fit the model for quadratic growth. 

A computer calculation gives 

“ 73.0701 70.1387~ 

[Pi ,fi 2 ]= 3.6444 4.0900 

|_ -2.0274 —1.8534_ 

so the estimated growth curves are 


Control group: 73.07 

+ 3.641 - 

2.03/ 2 

(2.58) 

(.83) 

(.28) 

Treatment group: 70.14 

+ 4.091 - 

1.85/ 2 

(2.50) 

(.80) 

(.27) 

[ 93.1744 

-5.8368 

0.2184 

S^oiedB) -1 = -5.8368 

9.5699 

-3.0240 

L 0.2184 

-3.0240 

1.1051 


and, by (6-79), the standard errors given below the parameter estimates were 
obtained by dividing the diagonal elements by n^ and taking the square root. 
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Examination of the estimates and the standard errors reveals that the t 2 terms 
are needed. Loss of calcium is predicted after 3 years for both groups. Further, there 
does not seem to be any substantial difference between the two groups. 

Wilks’ lambda for testing the null hypothesis that the quadratic growth model is 
adequate becomes 


2726.282 

2660.749 

2369.308 

2335.912 

2781.017 

2698.589 

2363.228 

2362.253 


2660.749 

2756.009 

2343.514 

2327.961 

2698.589 

2832.430 

2331.235 

2381.160 


2369.308 

2343.514 

2301.714 

2098.544 

2363.228 

2331.235 

2303.687 

2089.996 


2335.912 

2327.961 

2098.544 

2277.452 

2362.253 

2381.160 

2089.996 

2314.485 


= .7627 


Since, with a = .01, 


<? + g) 


-131 --(4-2 + 2) 


In .7627 


= 7.86 < *?4-2-l)2(-01) 


we fail to reject the adequacy of the quadratic fit at a = .01. Since the p -value is less 
than .05 there is, however, some evidence that the quadratic does not fit well. 

We could, without restricting to quadratic growth, test for parallel and coinci¬ 
dent calcium loss using profile analysis. ■ 

The Potthoff and Roy growth curve model holds for more general designs than 
one-way MANOVA. However, the are no longer given by (6-78) and the expres¬ 
sion for its covariance matrix becomes more complicated than (6-79). We refer the 
reader to [14] for more examples and further tests. 

There are many other modifications to the model treated here.They include the 
following: 

(a) Dropping the restriction to polynomial growth. Use nonlinear parametric 
models or even nonparametric splines. 

(b) Restricting the covariance matrix to a special form such as equally correlated 
responses on the same individual. 

(c) Observing more than one response variable, over time, on the same individual. 
This results in a multivariate version of the growth curve model. 


6.10 Perspectives and a Strategy for Analyzing 
Multivariate Models 

We emphasize that, with several characteristics, it is important to control the overall 
probability of making any incorrect decision. This is particularly important when 
testing for the equality of two or more treatments as the examples in this chapter 
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Example 6.16 (Comparing multivariate and univariate tests for »h. au 
means) Suppose we collect measurements on two variables X ,n 


*1 

*2 

Group 

5.0 

3.0 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

4.5 

3.2 

6.0 

3.5 

6.0 

4.6 

6.2 

5.6 

6.9 

6.8 

5.2 

6.0 

5.3 

5.5 

6.6 

7.3 

7.3 

, 6.5 

4.6 

4.9 

2 

4.9 

4.0 

5.9 

4.1 

2 

2 

3.8 

5.4 

2 

6.2 

6.1 

2 

5.0 

7.0 

2 

5.3 

4.7 

2 

7.1 

6.6 

2 

5.8 

7.8 

2 

6.8 

8.0 

2 
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«ess v « 

variate analysis of variance gives^F = 2 68 with v , 8 f * 2 ° 1 b o Servatl0ns ' a uni ‘ 
dom. Again" we cannot reject^. * 1 = 1 and ** = ^degrees offfee- 

J n 0 .fi 12 - fx 22 at any reasonable significance level. 
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Figure 6.6 Scatter plots and marginal dot diagrams for the data from two groups. 

The univariate tests suggest there is no difference between the component means 
for the two groups, and hence we cannot discredit Mi = M2 

On the other hand, if we use Hotelling’s T 2 to test for the equality of the mean 

vectors, we find 

T i = 17 29 > c 2 = = 2.118 x 6.11 = 12.94 


-F 2 i17 (.01) = 2.118 X 6.11 


and we reject H 0 : Ml = M 2 at the 1% level.The multivariate test takes into account 
the nositive correlation between the two measurements for each group—informa 
tion that is unfortunately ignored by the univariate tests. This T -test is equivalent to 
the MANOVA test (6-42). 

Fxamole 6.17 (Data on lizards that require a bivariate test to establish a difference in 
means) A zoologist collected lizards in the southwestern United States. Among 
other variables, he measured mass (in grams) and the snout-vent length (in millime¬ 
ters) Because the tails sometimes break off in the wild, the snout-vent length is a 
more representative measure of length. The data for the lizards from two genera, 
Cnemidophorus (C) and Sceloporus (S), collected in 1997 and 1999 are given in 
Table ^Notice that there are rii = 20 measurements for C lizards and n 2 - 40 
measurements for S lizards. 

After taking natural logarithms, the summary statistics are 

_ [2.240*1 _ f0.35305 0.09417 

C: ni =20 *1 - 4394 s i “ q.09417 0.02595 


0.50684 0.14539 
0.14539 0.04255 



F 2 . 24 ol 

C:«i = 20 

* i L4.394J 1 


[2.368*1 c 

S: n 2 = 40 

* 2 l 4 - 308 J 2 
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3.9 4.0 4,1 4.2 4.3 4.4 4,5 4.6 4.7 4.8 

ln(SVL) 

Figure 6.7 Scatter plot of ln(Mass) versus In(SVL) for the lizard data in Thble 6.7. 


A plot of mass (Mass) versus snout-vent length (SVL), after taking natural logarithms, 
is shown in Figure 6.7. The large sample individual 95% confidence intervals for the 
difference in ln(Mass) means and the difference in ln(SVL) means both cover 0. 

In(Mass): Mil “ M 2 i : (-0.476,0.220) 
ln(SVL): M 12 “ ^ (“0.011,0.183) 
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The corresponding univariate Student’s f-test statistics for testing for no difference 
in the individual means have p-values of .46 and .08, respectively. Clearly, from a 
univariate perspective, we cannot detect a difference in mass means or a difference 
in snout-vent length means for the two genera of lizards. 

However, consistent with the scatter diagram in Figure 6.7, a bivariate analysis 
strongly supports a difference in size between the two groups of lizards. Using Result 
6.4 (also see Example 6.5), the T 2 -statistic has an approximate xl distribution. 
For this example, T 2 = 225.4 with a p-value less than .0001. A multivariate method is 
essential in this case. ■ 

Examples 6.16 and 6.17 demonstrate the efficacy of a multivariate test relative 
to its univariate counterparts. We encountered exactly this situation with the efflu¬ 
ent data in Example 6.1. 

In the context of random samples from several populations (recall the one-way 
MANOVA in Section 6.4), multivariate tests are based on the matrices 

w = 2 (*o _ (*tj ~ *f)' and B = 1 - *) (*<■ - *)' 

e=i j= i e=1 

Throughout this chapter, we have used 

|W| 

Wilks’lambda statistic A* = . 

+ Wj 

which is equivalent to the likelihood ratio test. Three other multivariate test statis¬ 
tics are regularly included in the output of statistical packages. 

Lawley-Hotelling trace = t^BW^ 1 ] 

Pillai trace = tr[B(B + W) ] ] 

Roy’s largest root = maximum eigenvalue of W (B + W)” 1 

All four of these tests appear to be nearly equivalent for extremely large sam¬ 
ples. For moderate sample sizes, all comparisons are based on what is necessarily a 
limited number of cases studied by simulation. From the simulations reported to 
date, the first three tests have similar power, while the last, Roy’s test, behaves dif¬ 
ferently. Its power is best only when there is a single nonzero eigenvalue and, at the 
same time, the power is large. This may approximate situations where a large 
difference exists in just one characteristic and it is between one group and all of the 
others. There is also some suggestion that Pillai’s trace is slightly more robust 
against nonnormality. However, we suggest trying transformations on the original 
data when the residuals are nonnormal. 

All four statistics apply in the two-way setting and in even more complicated 
MANOVA. More discussion is given in terms of the multivariate regression model 
in Chapter 7. 

When, and only when, the multivariate tests signals a difference, or departure 
from the null hypothesis, do we probe deeper. We recommend calculating the 
Bonferonni intervals for all pairs of groups and all characteristics. The simultaneous 
confidence statements determined from the shadows of the confidence ellipse are, 
typically, too large.The one-at-a-time intervals may be suggestive of differences that 


Exercises 337 

merit further study but, with the current data, cannot be taken as conclusive evi¬ 
dence for the existence of differences. We summarize the procedure developed in 
this chapter for comparing treatments. The first step is to check the data for outliers 
using visual displays and other calculations. 


A Strategy for the Multivariate Comparison of Treatments 

1. Try to identify outliers. Check the data group by group for outliers. Also 
check the collection of residual vectors from any fitted model for outliers. 
JBe aware of any outliers so calculations can be performed with and without 
them. 

2. Perform a multivariate test of hypothesis. Our choice is the likelihood ratio 
test, which is equivalent to Wilks’ lambda test. 

3. Calculate the Bonferroni simultaneous confidence intervals . If the multi¬ 
variate test reveals a difference, then proceed to calculate the Bonferroni 
confidence intervals for all pairs of groups or treatments, and all character¬ 
istics. If no differences are significant, try looking at Bonferroni intervals for 
the larger set of responses that includes the differences and sums of pairs of 
responses. 

We must issue one caution concerning the proposed strategy. It may be the case 
that differences would appear in only one of the many characteristics and, further, 
the differences hold for only a few treatment combinations. Then, these few active 
differences may become lost among all the inactive ones. That is, the overall test may 
not show significance whereas a univariate test restricted to the specific active vari¬ 
able would detect the difference. The best preventative is a good experimental 
design. To design an effective experiment when one specific variable is expected to 
produce differences, do not include too many other variables that are not expected 
to show differences among the treatments. 


Exer cises _ 

6 . 1 . Construct and sketch a joint 95% confidence region for the mean difference vector 5 
using the effluent data and results in Example 6.1. Note that the point 5=0 falls 
outside the 95% contour. Is this result consistent with the test of H Q : 5 = 0 considered 
in Example 6.1? Explain. 

6.2. Using the information in Example 6.1. construct the 95% Bonferroni simultaneous in¬ 
tervals for the components of the mean difference vector 5. Compare the lengths of 
these intervals with those of the simultaneous intervals constructed in the example. 

6.3. The data corresponding to sample 8 in Thble 6.1 seem unusually large. Remove sample 8. 
Construct a joint 95% confidence region for the mean difference vector 8 and the 95% 
Bonferroni simultaneous intervals for the components of the mean difference vector. 
Are the results consistent with a test of H 0 : 8 = 0? Discuss. Does the “outlier” make a 
difference in the analysis of these data? 
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6.4. Refer to Example 6.1. 

(a) Redo the analysis in Example 6.1 after transforming the pairs of observations to 
In (BOD) and ln(SS). 

(b) Construct the 95% Bonferroni simultaneous intervals for the components of the 
mean vector 8 of transformed variables 

(c) Discuss any possible violation of the assumption of a bivariate normal distribution 
for the difference vectors of transformed observations. 

6.5. A researcher considered three indices measuring the severity of heart attacks. The 
values of these indices for n - 40 heart-attack patients arriving at a hospital emergency 
room produced the summary statistics 


x 


46.11 [101.3 63.0 71.0’ 

57.3 and S = 63.0 80.2 55.6 

50.4J • [ 71 ’° 55.6 97 * 4 - 


(a) All three indices are evaluated for each patient.Tfest for the equality of mean indices 
using (6-16) with a = .05. 

(b) Judge the differences in pairs of mean indices using 95% simultaneous confidence 
intervals. [See (6-18).] 

6 . 6 . Use the data for treatments 2 and 3 in Exercise 6.8. 

(a) Calculate S pooled . 

(b) Test H 0 : p 2 ~ M 3 = 0 employing a two-sample approach with a = .01. 

(c) Construct 99% simultaneous confidence intervals for the differences fi 2i ~ p 3i , 

i = l, 2 . 

6.7. Using the summary statistics for the electricity-demand data given in Example 6.4, com¬ 
pute T 2 and test the hypothesis H 0 \Mi ~ M 2 “ assuming that Xi = X 2 . Set a - .05. 
Also, determine the linear combination of mean components most responsible for the 
rejection of H 0 . 

6 . 8 . Observations on two responses are collected for three treatments. The obser- 

*1 

vation vectors 

_*2 

„ , M [si M [ 4 ] [7' 

Treatment 1: ^J, ^J. ^J. 



Treatment 2: 



^ „ [2*1 nn r 3 i [2' 

Treatment 3: ^ ^ , 

(a) Break up the observations into mean, treatment, and residual components, as in 
(6-39). Construct the corresponding arrays for each variable. (See Example 6.9.) 

(b) Using the information in Part a,construct the one-way MANOVA table. 

(c) Evaluate Wilks’ lambda, A*, and use Table 6.3 to test for treatment effects. Set 
a = .01. Repeat the test using the chi-square approximation with Bartlett’s correc¬ 
tion. [See (6-43).] Compare the conclusions. 
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6.9. Using the contrast matrix C in (6-13), verify the relationships d, = Cx f , d = Cx, and 

S, = CSC'in (6-14). r j j, 

6.10. Consider the univariate one-way decomposition of the observation x tj given by (6-34). 

Show that the mean vector x 1 is always perpendicular to the treatment effect vector 
(*1 ~ *) u i + (*2 ~ *) u 2 + + (3t^ — x )where 


6.11. A likelihood argument provides additional support for pooling the two independent 
sample covariance matrices to estimate a common covariance matrix in the case of two 
normal populations. Give the likelihood function, L(m 1 , M 2 > 2), for two independent 
samples of sizes and n 2 from A^(mi , X) and N p (p, 2 , X) populations, respectively. Show 
that this likelihood is maximized by the choices Mi = Xi, M 2 = x 2 and 


x - “ 1)Si + ( " 2 - 1)S2 J - 

Hint: Use (4-16) and the maximization Result 4.10. 

6.12. (Test for linear profiles ; given that the profiles are parallel) Let p\ = 
[M 11 > Mi 2 > • *■ y&i P ] and M 2 = [M 2 I 1 M 22 * • • • > M 2 p] be the mean responses to p treat¬ 
ments for populations 1 and 2, respectively. Assume that the profiles given by the two 
mean vectors are parallel. 

(a) Show that the hypothesis that the profiles are linear can be written as H 0 :(p li + M 2 /) ~ 
(M 1/-1 + M 2 /- 1 ) = + M 2 /- 1 ) ~ (Pn -2 + /X 2 /— 2 ). i = 3,...,p or as Hq: 

C (Mi + M 2 ) =0, where the (p - 2) X p matrix 


1-2 10 
0 1—2 1 


0 0 0 
0 0 0 


L° 0 0 0 1 -2 lj 

(b) Following an argument similar to the one leading to (6-73), we reject 
H 0 : C (mi + M 2 ) = 0 at level a if 


T 2 = (x r + x 2 )'C 


1 1 
— + — 


^^pooledC" 


C(xj + x 2 ) > 


c 2 = + ”2 - 2 ){ P - 2) 

„ I , . + D— 


n i + n 2 - p + 1 


p~2,n { + n 2 -p+\{ 


where 
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Let = 30,« 2 = 30, xi = [6.4,6.8,7.3,7.0], i' 2 = [43,4.9,5.3,5.1], and 

'.61 .26 .07 .16' 

.26 .64 .17 .14 

Spooled - 0? 1? gl 03 

.16 .14 .03 31. 


Test for linear profiles, assuming that the profiles are parallel. Use a = .05. 

6.13. (Two-way MANOVA without replications.) Consider the observations on two 
responses, jc, and x 2i displayed in the form of the following two-way table (note that 
there is a single observation vector at each combination of factor levels): 



With no replications, the two-way MANOVA model is 

g ^ 

X u = #1 + T( + Pt + e a ; 2 T e = 20* = O 

f=l *=*1 

where the e (k are independent A^(0, X) random vectors. 

(a) Decompose the observations for each of the two variables as 

x tk = x + ( x ( . - x) + (x. k - x) + (. x ik - x e . - x. k + *) 

similar to the arrays in Example 6.9. For each response , this decomposition will result 
in several 3X4 matrices. Here x is the overall average, x € . is the average for the f th 
level of factor 1 , and x . k is the average for the fcth level of factor 2. 

(b) Regard the rows of the matrices in Part a as strung out in a single “long” vector, and 
compute the sums of squares 

SS| 0 i — SS mean + SSf acl + SSf aC 2 + SS res 


and sums of cross products 

SCP tot = SCP mean + SCP facl + SCP fac2 + SCP res 

Consequently,obtain the matrices SSP cor , SSPf ac j, SSPf aC 2 , and SSP res with degrees 
of freedom gb -l,g-l,b- l,and (g - 1 ) (b - 1), respectively. 

(c) Summarize the calculations in Part b in a MANOVA table. 
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Hint: This MANOVA table is consistent with the two-way MANOVA table for com¬ 
paring factors and their interactions where n = 1 . Note that, with n ~ 1 , SSP res in the 
general two-way MANOVA table is a zero matrix with zero degrees of freedom. The 
matrix of interaction sum of squares and cross products now becomes the residual sum 
of squares and cross products matrix. 

(d) Given the summary in Part c, test for factor 1 and factor 2 main effects at the a = .05 
level. 

Hint: Use the results in (6-67) and (6-69) with gb(n — 1) replaced by (g — 1) (b ~ 1)- 
Note: The tests require that p ^ (g - 1) (b - 1) so that SSP res will be positive defi¬ 
nite (with probability 1). 

6.14. A replicate of the experiment in Exercise 6.13 yields the following data: 



(a) Use these data to decompose each of the two measurements in the observation 
vector as 

X(k = x + (*f. - S) + (i-t - S) + (*f* - if. - i.* + i) 

where x is the overall average, x € . is the average for the £th level of factor 1 , and x . k 
is the average for the fcth level of factor 2. Form the corresponding arrays for each of 
the two responses. 

(b) Combine the preceding data with the data in Exercise 6.13 and carry out the neces¬ 
sary calculations to complete the general two-way MANOVA table. 

(c) Given the results in Part b, test for interactions, and if the interactions do not 
exist, test for factor 1 and factor 2 main effects. Use the likelihood ratio test with 
a = .05. 

(d) If main effects, but no interactions, exist, examine the nature of the main effects by 
constructing Bonferroni simultaneous 95% confidence intervals for differences of 
the components of the factor effect parameters. 

6.15. Refer to Example 6.13. 

(a) Carry out approximate chi-square (likelihood ratio) tests for the factor 1 and factor 2 
effects. Set a = .05. Compare these results with the results for the exact F-tests given 
in the example. Explain any differences. 

(b) Using (6-70), construct simultaneous 95% confidence intervals for differences in the 
factor 1 effect parameters for pairs of the three responses. Interpret these intervals. 
Repeat these calculations for factor 2 effect parameters. 
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The following exercises may require the use of a computer. 

6. 16 , Four measures of the response stiffness on each of 30 boards are listed in Table 4.3 (see 
Example 4.14).The measures, on a given board, are repeated in the sense that they were 
made one after another. Assuming that the measures of stiffness arise from four 
treatments, test for the equality of treatments in a repeated measures design context. Set 
a — .05. Construct a 95% (simultaneous) confidence interval for a contrast in the 
mean levels representing a comparison of the dynamic measurements with the static 
measurements. 

6.1,7. The data in Table 6.8 were collected to test two psychological models of numerical 
cognition. Does the processing of numbers depend on the way the numbers are pre¬ 
sented (words, Arabic digits)? Thirty-two subjects were required to make a series of 


Table 6.B Number Parity Data (Median Times in Milliseconds) 


WordDiff 

(* 1 ) 

WordS ame 

(* 4 ) 

ArabicDiff 

to) 

ArabicSame 

(* 4 ) 

869.0 

860.5 

691.0 

601.0 

995.0 

875.0 

678.0 

659.0 

1056.0 

930.5 

833.0 

826.0 

1126.0 

954.0 

888.0 

728.0 

1044.0 

909.0 

865.0 

839.0 

925.0 

856.5 

1059.5 

797.0 

1172.5 

896.5 

926.0 

766.0 

1408.5 

1311.0 

854.0 

986.0 

1028.0 

887.0 

915.0 

735.0 

1011.0 

863.0 

761.0 

657.0 

726.0 

674.0 

663.0 

583.0 

982.0 

894.0 

831.0 

640.0 

1225.0 

1179.0 

1037.0 

905.5 

731.0 

662.0 

662.5 

624.0 

975.5 

872.5 

814.0 

735.0 

1130.5 

811.0 

843.0 

657.0 

945.0 

909.0 

867.5 

754.0 

747.0 

752.5 

777.0 

, 687.5 

656.5 

659.5 

572.0 

539.0 

919.0 

833.0 

752.0 

611.0 

751.0 

744.0 

683.0 

553.0 

774.0 

735.0 

671.0 

612.0 

941.0 

931.0 

901.5 

700.0 

751.0 

785.0 

789.0 

735.0 

767.0 

737.5 

724.0 

639.0 

813.5 

750.5 

711.0 

625.0 

1289.5 

1140.0 

904.5 

784.5 

1096.5 

1009.0 

1076.0 

983.0 

1083.0 

958.0 

918.0 

746.5 

1114.0 

1046.0 

1081.0 

796.0 

708.0 

669.0 

657.0 

572.5 

1201.0 

925.0 

1004.5 

673.5 

Source: Data courtesy of J. Carr. 
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quick numerical judgments about two numbers presented as either two number 
words (“two ” “four”) or two single Arabic digits (“2,” “4”). The subjects were asked 
to respond same” if the two numbers had the same numerical parity (both even or 
both odd) and different” if the two numbers had a different parity (one even, one 
odd). Half of the subjects were assigned a block of Arabic digit trials, followed by a 
block of number word trials, and half of the subjects received the blocks of trials 
in the reverse order. Within each block, the order of “same” and “different” parity 
trials was randomized for each subject. For each of the four combinations of parity and 
format, the median reaction times for correct responses were recorded for each 
subject. Here 

X\ = median reaction time for word format-different parity combination 
X 2 — median reaction time for word format—same parity combination 
X^ — median reaction time for Arabic format-different parity combination 
X A = median reaction time for Arabic format-same parity combination 

(a) Test for treatment effects using a repeated measures design. Set a = .05. 

(b) Construct 95% (simultaneous) confidence intervals for the contrasts representing 
the number format effect, the parity type effect and the interaction effect. Interpret 
the resulting intervals. 

(c) The absence of interaction supports the M model of numerical cognition, while the 
presence of interaction supports the C and C model of numerical cognition. Which 
model is supported in this experiment? 

(d) For each subject, construct three difference scores corresponding to the number for¬ 
mat contrast, the parity type contrast, and the interaction contrast. Is a multivariate 
normal distribution a reasonable population model for these data? Explain. 

6 . 18 . Jolicoeur and Mosimann [12] studied the relationship of size and shape for painted tur¬ 
tles. Thble 6.9 contains their measurements on the carapaces of 24 female and 24 male 
turtles. 

(a) Test for equality of the two population mean vectors using a = .05. 

(b) If the hypothesis in Part a is rejected, find the linear combination of mean compo¬ 
nents most responsible for rejecting H 0 . 

(c) Find simultaneous confidence intervals for the component mean differences. 
Compare with the Bonferroni intervals. 

Hint: You may wish to consider logarithmic transformations of the observations. 

6.19. In the first phase of a study of the cost of transporting milk from farms to dairy plants, a 
survey was taken of firms engaged in milk transportation. Cost data on Xi = fuel, 
^ 2 “ repair, and X$ — capital, all measured on a per-mile basis, are presented in 
Table 6.10 on page 345 for m = 36 gasoline and n 2 = 23 diesel trucks. 

(a) Test for differences in the mean cost vectors. Set a = .01. 

(b) If the hypothesis of equal cost vectors is rejected in Part a, find the linear combina¬ 
tion of mean components most responsible for the rejection. 

(c) Construct 99% simultaneous confidence intervals for the pairs of mean components. 
Which costs, if any, appear to be quite different? 

(d) Comment on the validity of the assumptions used in your analysis. Note in particular 
that observations 9 and 21 for gasoline trucks have been identiGed as multivariate 
outliers. (See Exercise 5.22 and [2].) Repeat Part a with these observations deleted. 
Comment on the results. 
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Table 6.9 Carapace Measurements (in Millimeters) for 

Painted Turtles 

Female 

Male 

Length 

(*i)' 

Width 

(■* 2 ) 

Height 

(•* 3 ) 

Length 

(■* 1 ) 

Width 

(* 2 ) 

Height 

(* 3 ) 

98 

81 

38 

93 

74 

37 

103 

84 

38 

94 

78 

35 

103 

86 

42 

96 

80 

35 

105 

86 

42 

101 

84 

39 

109 

88 

44 

102 

85 

38 

123 

92 

50 

103 

81 

37 

123 

95 

46 

104 

83 

39 

133 

99 

51 

106 

83 

39 

133 

102 

51 

107 

82 

38 

133 

102 

51 

112 

89 

40 

134 

100 

48 

113 

88 

40 

136 

102 

49 

114 

86 

40 

138 

98 

51 

116 

90 

43 

138 

99 

51 

117 

90 

41 

141 

105 

53 

117 

91 

41 

147 

108 

57 

119 

93 

41 

149 

107 

55 

120 

89 

40 

153 

107 

56 

120 

93 

44 

155 

115 

63 

121 

95 

42 

155 

117 

60 

125 

93 

45 

158 

115 

62 

127 

96 

45 

159 

118 

63 

128 

95 

45 

162 

124 

61 

131 

95 

46 

177 

132 

67 

135 

106 

47 




6.20. The tail lengths in millimeters (*i) and wing lengths in millimeters ( x 2 ) for 45 male 
hook-billed kites are given in Thble 6.11 on page 346. Similar measurements for female 
hook-billed kites were given in Table 5.12. 

(a) Plot the male hook-billed kite data as a scatter diagram, and (visually) check for out¬ 
liers. (Note,in particular, observation 31 with x\ — 284.) 

(b) Test for equality of mean vectors for the populations of male and female hook¬ 
billed kites. Set a = .05. If H Q : fii - fi 2 ~ 0 is rejected, find the linear combina¬ 
tion most responsible for the rejection of H 0 . (You may want to eliminate any 
outliers found in Part a for the male hook-billed kite data before conducting this 
test. Alternatively, you may want to interpret Xi = 284 for observation 31 as a mis¬ 
print and conduct the test with jcj = 184 for this observation. Does it make any 
difference in this case how observation 31 for the male hook-billed kite data is 
treated?) 

(c) Determine the 95% confidence region for - fi 2 and 95% simultaneous confi¬ 
dence intervals for the components of - fi 2 • 

(d) Are male or female birds generally larger? 
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Table 6.10 Milk Transportation-Cost Data 

Gasoline trucks 

Diesel trucks 

*1 

*2 

*3 

*1 

* 2 

*3 

16.44 

12.43 

11.23 

8.50 

12.26 

9.11 

7.19 

2.70 

3.92 

7.42 

5.13 

17.15 

9.92 

1.35 

9.75 

10.28 

3.32 

11.23 

4.24 

5.78 

7.78 

10.16 

14.72 

5.99 

11.20 

5.05 

10.67 

12.79 

4.17 

29.28 

14.25 

5.78 

9.88 

9.60 

12.72 

11.00 

13.50 

10.98 

10.60 

6.47 

8.89 

19.00 

13.32 

14.27 

9.45 

11.35 

9.95 

14.53 

29.11 

15.09 

3.28 

9.15 

2.94 

13.68 

12.68 

7.61 

10.23 

9.70 

5.06 

20.84 

7.51 

5.80 

8.13 

9.77 

17.86 

35.18 

9.90 

3.63 

9.13 

11.61 

11.75 

17.00 

10.25 

5.07 

10.17 

9.09 

13.25 

20.66 

11.11 

6.15 

7.61 

8.53 

10.14 

17.45 

12.17 

14.26 

14.39 

8.29 

6.22 

16.38 

10.24 

2.59 

6.09 

15.90 

12.90 

19.09 

10.18 

6.05 

12.14 

11.94 

5.69 

14.77 

8.88 

2.70 

12.23 

9.54 

16.77 

22.66 

12.34 

7.73 

11.68 

10.43 

17.65 

10.66 

8.51 

14.02 

12.01 

10.87 

21.52 

28.47 

26.16 

17.44 

16.89 

7.13 

13.22 

19.44 

12.95 

8.24 

7.18 

11.88 

12.18 

21.20 

16.93 

13.37 

17.59 

12.03 

9.22 

23.09 

14.70 

10.78 

14.58 




10.32 

5.16 

17.00 




8.98 

4.49 

4.26 




9.70 

11.59 

6.83 




12.72 

8.63 

5.59 




9.49 

2.16 

6.23 




8.22 

7.95 

6.72 




13.70 

11.22 

4.91 




8.21 

9.85 

8.17 




15.86 

11.42 

13.06 




9.18 

9.18 

9.49 




12.49 

4.67 

11.94 




17.32 

6.86 

4.44 




Source: Data courtesy of M. Keaton. 


6.21. Using Moody’s bond ratings, samples of 20 Aa (middle-high quality) corporate bonds 
and 20 Baa (top-medium quality) corporate bonds were selected. For each of the corre¬ 
sponding companies, the ratios 

X x = current ratio (a measure of short-term liquidity) 

X 2 = long-term interest rate (a measure of interest coverage) 

X 3 ~ debt-to-equity ratio (a measure of financial risk or leverage) 

X 4 ~ rate of return on equity (a measure of profitability) 
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(c) Calculate the linear combinations of mean components most responsible for rejecting 
H q : fii - fi 2 - 0 in Part b. 

(d) Bond rating companies are interested in a company’s ability to satisfy its outstanding 
debt obligations as they mature. Does it appear as if one or more of the foregoing 
financial ratios might be useful in helping to classify a bond as “high” or “medium” 
quality? Explain. 

(e) Repeat part (b) assuming normal populations with unequal covariance matices (see 
(6-27), (6-28) and (6-29)). Does your conclusion change? 

6.22. Researchers interested in assessing pulmonary function in nonpathological populations 
asked subjects to run on a treadmill until exhaustion. Samples of air were collected at 
definite intervals and the gas contents analyzed. The results on 4 measures of oxygen 
consumption for 25 males and 25 females are given in Table 6.12 on page 348. The 
variables were 

Xi = resting volume 0 2 (L/min) 

X 2 — resting volume 0 2 (mL/kg/min) 

X 3 ~ maximum volume 0 2 (L/min) 

X$ = maximum volume 0 2 (mL/kg/min) 

(a) Look for gender differences by testing for equality of group means. Use a = .05. If 

you reject H 0 : - /i 2 = 0, find the linear combination most responsible. 

(b) Construct the 95% simultaneous confidence intervals for each /i*, - \x . 2l , i = 1,2,3,4. 
Compare with the corresponding Bonferroni intervals. 

(c) The data in Thble 6.12 were collected from graduate-student volunteers, and thus 
they do not represent a random sample. Comment on the possible implications of 
this information. 

6.23. Construct a one-way MANOVA using the width measurements from the iris data in 
Table 11.5. Construct 95% simultaneous confidence intervals for differences in mean 
components for the two responses for each pair of populations. Comment on the validity 
of the assumption that Xi = X 2 = X 3 . 

6.24. Researchers have suggested that a change in skull size over time is evidence of the inter¬ 
breeding of a resident population with immigrant populations. Four measurements were 
made of male Egyptian skulls for three different time periods: period 1 is 4000 EC, period 2 
is 3300 B.c., and period 3 is 1850 B.c. The data are shown in Table 6.13 on page 349 (see the 
skull data on the website www.prenhall.com/statistics). The measured variables are 

Xi - maximum breadth of skull (mm) 

X 2 - basibregmatic height of skull (mm) 

X 3 = basialveolar length of skull (mm) 

X 4 — nasal height of skull (mm) 

Construct a one-way MANOVA of the Egyptian sl^ull data. Use a = .05. Construct 95 % 
simultaneous confidence intervals to determine which mean components differ among 
the populations represented by the three time periods Are the usual MANOVA as¬ 
sumptions realistic for these data? Explain. 

6.25. Construct a one-way MANOVA of the crude-oil data listed in Thble 11.7 on page 662. 
Construct 95% simultaneous confidence intervals to determine which mean compo¬ 
nents differ among the populations (You may want to consider transformations of the 
data to make them more closely conform to the usual MANOVA assumptions) 
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Table 6.13 Egyptian Skull Data 

MaxBreath 

(*i) 

BasHeight 

(x 2 ) 

BasLength 

to) 

NasHeight 

(*4) 

Time 

Period 

131 

138 

89 

49 

1 

125 

131 

92 

48 

1 

131 

132 

99 

50 

1 

119 

132 

96 

44 

1 

136 

143 

100 

54 

1 

138 

137 

89 

56 

1 

139 

130 

108 

48 

1 

125 

136 

93 

48 

1 

131 

134 

102 

51 

1 

134 

134 

99 

51 

1 

124 

138 

101 

48 

2 

133 

134 

97 

48 

2 

138 

134 

98 

45 

2 

148 

129 

104 

51 

2 

126 

124 

95 

45 

2 

135 

136 

98 

52 

2 

132 

145 

100 

54 

2 

133 

130 

102 

48 

2 

131 

134 

96 

50 

2 

133 

125 

94 

46 

2 

132 

130 

91 

52 

3 

133 

131 

100 

50 

3 

138 

137 

94 

51 

3 

130 

127 

99 

45 

3 

136 

133 

91 

49 

3 

134 

123 

95 

52 

'3 

136 

137 

101 

54 

3 

133 

131 

96 

49 

3 

138 

133 

100 

55 

3 

138 

133 

91 

46 

3 

Source: Data courtesy of J. Jackson. 


6.26. A project was designed to investigate how consumers in Green Bay, Wisconsin, would 
react to an electrical time-of-use pricing scheme. The cost of electricity during peak 
periods for some customers was set at eight times the cost of electricity during 
off-peak hours. Hourly consumption (in kilowatt-hours) was measured on a hot summer 
day in July and compared, for both the test group and the control group, with baseline 
consumption measured on a similar day before the experimental rates began. The 
responses, 

log(current consumption) - log(baseline consumption) 
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for the hours ending9 a.m. 11a.m. (a peak hour), 1 P.M.,and 3 P.M. (a peak hour) produced 
the following summary statistics: 


Test group: 
Control group: 
and 


= 28,*', = [153, -.231, -.322, -339] 
n 2 = 58, *2 = [.151, .180, 256,257} 


'.804 355 328 .232 
355 .722 .233 .199 
Spooled - 228 .233 .592 .239 

.232 .199 .239 .479, 

^rceToata courtesy of Statistical Laboratory, University of Wisconsin. 

d a profile analysis. Does time-of-use pricing seem to make a difference in 

electrical consumption? What is the nature of this difference, if any? Comment. (Use a 
significance level of a - 05 for any statistical tests.) 

6 27 As Dart of the study of love and marriage in Example 6.14, a sample of husbands and 
‘ ’ wives were asked to respond to these questions: 

1 What is the level of passionate love you feel for your partner? 

2 What is the level of passionate love that your partner feels for you? 

3 What is the level of companionate love that you feel for your partner? 

4 What is the level of companionate love that your partner feels for you? 

The responses were recorded on the following 5-point scale. 

A peat Tremendous 

None ..y Some deal amount 


Thirty husbands and 30 wives gave the responses in Table 6.14, where X, - a 5-pornt- 
5e response to Question 1, X 2 = a 5-point-scale response to Question 2, * 3 - a 
^point-scale response to Question 3, and X A - a 5-point-scale response to Question 4. 

fa) Plot the mean vectors for husbands and wives as sample profiles, 
hi Is the husband rating wife profile parallel to the wife rating husband profile? Test 
b for Darallel profiles with a = .05. if the profiles appear to be parallel, test for coin¬ 
cident profiles at the same level of significance. Finally if the profiles are coinci¬ 
dent, test for level profiles with a = .05. What conclusion(s) can be drawn from this 

analysis? 

* 28 Two snecies of biting flies (genus Leptoconops) are so similar morphologically, that for 
6 ‘ 28 ' mnnv vears they were thou Jit to be the same. Biological differences such as sex ratios of 
probtp flies <md biting habits were found to exist. Do the taxonomic data listed in part 
em T JL g 615 on page 352 and on the website www.prenhall.com/statistics indicate any 

d \. mp;ln vectors using a = .05. If the hypotheses of equal mean vectors is rejected, 
dp 3 ,ermine the mean components (or linear combinations of mean components) mos 
S3 k *“*> *» - mediod, to, .he* d.». 

6.29. Using ihe dita on boon miner.! conient in T.ble 1.8. inveslig.le equality belween Ihe 
dominant and nondominant bones. 
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Table 6.14 Spouse Data 

Husband rating wife 

Wife rating husband 

-*i 

*2 

' *3 

x 4 

-*i 

x 2 

*3 

x 4 

2 

3 

5 

5 

4 

4 

5 

5 

5 

5 

4 

4 

4 

5 

5 

5 

4 

5 

5 

5 

4 

4 

5 

5 

4 

3 

4 

4 

4 

5 

5 

5 

3 

3 

5 

5 

4 

4 

5 

5 

3 

3 

4 

5 

3 

3 

4 

4 

3 

4 

4 

4 

4 

3 

5 

4 

4 

4 

5 

5 

3 

4 

5 

5 

4 

5 

5 

5 

4 

4 

5 

4 

4 

4 

3 

3 

3 

4 

4 

4 

4 

4 

5 

5 

4 

5 

5 

5 

5 

5 

4 

4 

5 

5 

5 

5 

4 

4 

4 

4 

4 

4 

5 

5 

4 

3 

5 

5 

4 

4 

4 

4 

4 

4 

5 

5 

4 

4 

5 

5 

3 

3 

4 

5 

3 

4 

4 

4 

4 

5 

4 

4 

5 

5 

5 

5 

5 

5 

5 

5 

4 

5 

4 

4 

5 

5 

4 

4 

3 

4 

4 

4 

4 

4 

4 

4 

5 

3 

4 

4 

4 

4 

4 

4 

5 

3 

4 

4 

4 

4 

4 

4 

4 

5 

4 

4 

3 

4 

5 

5 

2 

5 

5 

5 

5 

3 

5 

5 

3 

4 

5 

5 

5 

5 

3 

3 

4 

3 

5 

5 

3 

3 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

4 

5 

5 

3 

3 

5 

5 

3 

4 

4 

4 i 

4 

4 

3 

3 

4 

4 

5 

4 

4 

4 

5 

5 

4 

4 

5 

5 

Source: Data courtesy of E. Hatfield. 


(a) Test using a = .05. 

(b) Construct 95% simultaneous confidence intervals for the mean differences. 

(c) Construct the Bonferroni 95% simultaneous intervals, and compare these with the 
intervals in Part b. 

6.30. Table 6.16 on page 353 contains the bone mineral contents, for the first 24 subjects in 
Thble 1.8,1 year after their participation in an experimental program. Compare the data 
from both tables to determine whether there has been bone loss. 

(a) Test using a = .05. 

(b) Construct 95% simultaneous confidence intervals for the mean differences. 

(c) Construct the Bonferroni 95% simultaneous intervals, and compare these with the 
intervals in Part b. 
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Table 6.16 Mineral Content 

in Bones (After 1 Year) 




Subject 

number 

Dominant 

radius 

Radius 

Dominant 

humerus 

Humerus 

Dominant 

ulna 

Ulna 

1 

1.027 

1.051 

2.268 

2.246 

.869 

.964 

2 

.857 

.817 

1.718 

1.710 

.602 

.689 

3 

.875 

.880 

1.953 

1.756 

.765 

.738 

4 

.873 

.698 

1.668 

1.443 

.761 

.698 

5 

.811 

.813 

1.643 

1.661 

.551 

.619 

6 

.640 

.734 

1.396 

1.378 

.753 

.515 

7 

.947 

.865 

1.851 

1.686 

.708 

.787 

8 

.886 

.806 

1.742 

1.815 

.687 

.715 

9 

.991 

.923 

1.931 

1.776 

.844 

.656 

10 

.977 

.925 

1.933 

2.106 

.869 

.789 

11 

.825 

.826 

1.609 

1.651 

.654 

.726 

12 

.851 

.765 

2.352 

1.980 

.692 

.526 

13 

.770 

.730 

1.470 

1.420 

.670 

.580 

14 

.912 

.875 

1.846 

1.809 

.823 

.773 

15 

.905 

.826 

1.842 

1.579 

.746 

.729 

16 

.756 

.727 

1.747 

1.860 

.656 

.506 

17 

.765 

.764 

1.923 

1.941 

.693 

.740 

18 

.932 

.914 

2.190 

1.997 

.883 

.785 

19 

.843 

.782 

1.242 

1.228 

.577 

.627 

20 

.879 

.906 

2.164 

1.999 

.802 

.769 

21 

.673 

.537 

1.573 

1.330 

.540 

.498 

22 

.949 

.900 

2.130 

2.159 

.804 

.779 

23 

.463 

.637 

1.041 

1.265 

.570 

.634 

24 

.776 

.743 

1.442 

1.411 

.585 

.640 

Source: Data courtesy of Everett Smith. 


6.3 1 . Peanuts are an important crop in parts of the southern United States. In an effort to de¬ 
velop improved plants, crop scientists routinely compare varieties with respect to sever¬ 
al variables. The data for one two-factor experiment are given in Table 6.17 on page 354. 
Three varieties (5,6, and 8) were grown at two geographical locations (1,2) and, in this 
case, the three variables representing yield and the two important grade-grain charac¬ 
teristics were measured. The three variables are 

Xi = Yield (plot weight) 

X 2 = Sound mature kernels (weight in grams—maximum of 250 grams) 

X 3 = Seed size (weight, in grams, of 100 seeds) 

There were two replications of the experiment. 

(a) Perform a two-factor MANOVA using the data in Table 6.17. Tfest for a location 
effect, a variety effect, and a location-variety interaction. Use a = .05. 

(b) Analyze the residuals from Part a. Do the usual MANOVA assumptions appear to 
be satisfied? Discuss. 

(c) Using the results in Part a, can we conclude that the location and/or variety effects 
are additive? If not, does the interaction effect show up for some variables, but not 
for others? Check by running three separate univariate two-factor ANOVAs. 
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Table 6.17 Peanut Data 

Factor 1 
Location 

Factor 2 
Variety 

*1 

Yield 

* 2 

SdMatKer 

x 3 ' 

SeedSize 

1 

5 

195.3 

153.1 

51.4 

1 

5 

194.3 

167.7 

53.7 

2 

5 

189.7 

139.5 

555 

2 

5 

180.4 

121.1 

44.4 

1 

6 

203.0 

156.8 

49.8 

1 

6 

195.9 

166.0 

45.8 

2 

6 

202.7 

166.1 

60.4 

2 

6 

197.6 

161.8 

54.1 

1 

8 

193.5 

164.5 

57.8 

1 

8 

187.0 

165.1 

58.6 

2 

8 

201.5 

166.8 

65.0 

2 

8 

200.0 

173.8 

67.2 

Source . Data courtesy of Yolanda Lopez. 


(d) Larger numbers correspond to better yield and grade-grain characteristics. Using lo-g 
cation 2, can we conclude that one variety is better than the other two for each char|| 
acteristic? Discuss your answer, using 95% Bonferroni simultaneous intervals forj 
pairs of varieties. 

6.32. In one experiment involving remote sensing, the spectral reflectance of three species o| 
1-year-old seedlings was measured at various wavelengths during the growing seasonS 
The seedlings were grown with two different levels of nutrient: the optimal leveljp 
coded + , and a suboptimal level, coded -. The species of seedlings used were s ^ii 
spruce (SS), Japanese larch (JL), and lodgepole pine (LP). T\vo of the variables mea^P 
sured were 

X\ = percent spectral reflectance at wavelength 560 nm (green) 

X 2 — percent spectral reflectance at wavelength 720 nm (near infrared) 

The cell means (CM) for Julian day 235 for each combination of species and nutrient ^^ 
level are as follows. These averages are based on four replications. 


560CM 

720CM 

Species 

Nutrient 

10.35 

25.93 

SS 

+ 

13.41 

38.63 

JL 

+ 

7.78 

25.15 

LP 

+ 

10.40 

24.25 

SS 

- 

17.78 

41.45 

JL 

- 

10.40 

29.20 

LP 

— 


(a) 


Treating the cell means as individual observations, perform a two-way MANOV^|| 
test for a species effect and a nutrient effect. Use a = .05. w|| 

(b) Construct a two-way ANOVA for the 560CM observations and another two-\g| 
ANOVA for the 720CM observations. Are these results consistent wi 
MANOVA results in Part a? If not, can you explain any differences? 
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6.33. Refer to Exercise 6.32. The data in Table 6.18 are measurements on the variables 
X\ = percent spectral reflectance at wavelength 560 nm (green) 

Xi = percent spectral reflectance at wavelength 720 nm (near infrared) 

for three species (sitka spruce [SS], Japanese larch [JL], and lodgepole pine [LP]) of 
1-year-old seedlings taken at three different times (Julian day 150 [1], Julian day 235 [2], 
and Julian day 320 [3]) during the growing season. The seedlings were all grown with the 
optimal level of nutrient. 

(a) Perform a two-factor MANOVA using the data in Table 6.18. Test for a species 
effect, a time effect and species-time interaction. Use a = .05. 


Table 6.18 Spectral Reflectance Data 

560 nm 

720 nm 

Species 

Time 

Replication 

9.33 

19.14 

SS 

1 

1 

8.74 

19.55 

SS 

1 

2 

9.31 

19.24 

SS 

1 

3 

8.27 

16.37 

SS 

1 

4 

10.22 

25.00 

SS 

2 

1 

10.13 

25.32 

SS 

2 

2 

10.42 

27.12 

SS 

2 

3 

10.62 

26.28 

SS 

2 

4 

15.25 

38.89 

SS 

3 

1 

16.22 

36.67 

SS 

3 

2 

17.24 

40.74 

SS 

3 

3 

12.77 

67.50 

SS 

3 

4 

12.07 

33.03 

JL 

1 

1 

11.03 

32.37 

JL 

1 

2 

12.48 

31.31 

JL 

1 

3 

12.12 

33.33 

JL 

1 

4 

15.38 

40.00 

JL 

2 

1 

14.21 

40.48 

JL 

2 

2 

9.69 

33.90 

JL 

2 

3 

14.35 

40.15 

JL 

2 

4 

38.71 

77.14 

JL 

3 

1 

44.74 

78.57 

JL 

3 

2 

36.67 

71.43 

JL 

3 

3 

37.21 

45.00 

JL 

3 

4 

8.73 

2327 

LP 

1 

1 

7.94 

20.87 

LP 

1 

2 

8.37 

22.16 

LP 

1 

3 

7.86 

21.78 

LP 

1 

4 

8.45 

26.32 

LP 

2 

1 

6.79 

22.73 

LP 

2 

2 

8.34 

26.67 

LP 

2 

3 

7.54 

24.87 

LP 

2 

4 

14.04 

44.44 

LP 

3 

1 

13.51 

37.93 

LP 

3 

2 

13.33 

37.93 

LP 

3 

3 

12.77 

60.87 

LP 

3 

4 

Source: Data courtesy of Mairtin Mac Siurtain. 
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(b) S— 

vacate twofactor ANOVA for each of the two responses. 

numbers? ; s 

614 ssss.—- *.—- - - --—4 

the same graph. Comment on the comparison. | 

fht Test that linear growth is adequate. Take a — -01. 

6.35. Refer to Example 6.15 but treat all 31 subjects as a single group V* tnax.mum hkeg 
hood estimate of the (g + 1) x 1 P IS -~s 

p = (B’S" l B) -1 B'S" 1 X ^ 

Wh Ue!sSE^ like,ihood eStimat0tS ^ jp 

__ « _ (w - 1) [n - 2 )_ -(B’S^B) -1 i 

Cov(P) ~ p + q){n - p + q)n % 

m . quadratic grow.h cur.c .0 Ibis angle group art comment on .be Ct « 

6.36. Refer to Example 6.4. Given the »mm«ry mtonnatioj. on i „ cov 

pie, use Box’s M-test to test the ^ of Wisconsin homeowne^ 

of Wisconsin homeowners without a.r conditioning. ^ ma , e 

6.37. Table 6.9 page 344 contains the carapacemeasurements ^0^24 CO variance. 

ties. Use Box’s Af-test to test #o_ ^ ^ d s ; s the population covar^p 

matrix for carapace measurements for female tumes, an 2 ^ 

S matrix ^carapace measurements for male turtle. Set a - ^ ^ ^ 

6 38 Table 117 page 662 contains the values of three trace ele ^ nts ^ f san dstone. Useful 
drocarbons for crude oil samples taken from three S rou P s . ^"“{h, three sandstonj^? 

Box’s Af-test to test equality of population covanance main consider uans^^ 

groups'set <x = .05. Here there are p = 5 vartabl^ and you may w^^ ^ 
formations of the measurements on these vanables to make th m ^ ^ ^ 

6.39. Anacondas are some of the largest (cm) or the 

-• - ° 5 - Ap f 

large sample statistic. ^ 

(b) Is it reasonable to pool variances in this case? Explaim differences bet 

(c) Find the 95% Boneferrom confidence intervals fo 
males and females on both length and weight 
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Table 6.19 

Anaconda Data 





Snout vent 



Snout vent 



Length 

Weight Gender 

length 

Weight 

Gender 

271.0 

18.50 

F 

176.7 

3.00 

M 

477.0 

82.50 

F 

259.5 

9.75 

M 

306.3 

23.40 

F 

258.0 

10.07 

M 

365.3 

33.50 

F 

229.8 

7.50 

M 

466.0 

69.00 

F 

233.0 

6.25 

M 

440.7 

54.00 

F 

237.5 

9.85 

M 

315.0 

24.97 

F 

268.3 

10.00 

M 

417.5 

56.75 

F 

222.5 

9.00 

M 

307.3 

23.15 

F 

186.5 

3.75 

M 

319.0 

29.51 

F 

238.8 

9.75 

M 

303.9 

19.98 

F 

257.6 

9.75 

M 

331.7 

24.00 

F 

172.0 

3.00 

M 

435.0 

70.37 

F 

244.7 

10.00 

M 

261.3 

15.50 

F 

224.7 

7.25 

M 

384.8 

63.00 

F 

231.7 

9.25 

M 

360.3 

39.00 

F 

235.9 

7.50 

M 

441.4 

53.00 

F 

236.5 

5.75 

M 

246.7 

15.75 

F 

247.4 

7.75 

M 

365.3 

44.00 

F 

223.0 

5.75 

M 

336.8 

30.00 

F 

223.7 

5.75 

M 

326.7 

34.00 

F 

212.5 

7.65 

M 

312.0 

25.00 

F 

223.2 

7.75 

M 

226.7 

9.25 

F 

225.0 

5.84 

M 

347.4 

30.00 

F 

228.0 

7.53 

M 

280.2 

15.25 

F 

215.6 

5.75 

M 

290.7 

21.50 

F 

221.0 

6.45 

M 

438.6 

57.00 

F 

236.7 

6.49 

M 

377.1 

61.50 

F 

235.3 

6.00 

M 

Source: Data Courtesy of Jesus Ravis. 


6.40. Compare the male national track records in Table 8.6 with the female national track 
records in Table 1.9 using the results for the 100m, 200m, 400m, 800m and 1500m races. 
Treat the data as a random sample of size 64 of the twelve record values. 

(a) Test for equality of means between males and females using a = .05. Explain why it 
may be appropriate to analyze differences. 

(b) Find the 95% Bonferroni confidence intervals for the mean differences between 
male and females on all of the races. 

6«4I. When cell phone relay towers are not working properly, wireless providers can lose great 
amounts of money so it is important to be able to fix problems expeditiously. A first step 
toward understanding the problems involved is to collect data from a designed experi¬ 
ment involving three factors. A problem was initially classified as low or high severity, 
simple or complex, and the engineer assigned was rated as relatively new (novice) or 
expert (guru). 
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TWo times were observed. The time to assess the problem and plan an attack andj 
the time to implement the solution were each measured in hours. The data are given in* 

^ Pe 3 4 5 6 7 8 rform a MANOVA including appropnate confidence intervals for important effect* 


Table 6. 20 Fixing Breakdowns _ 

Problem Problem Engineer 

Severity Complexity Experience 

Level Level Level 

Low Simple Novice 

Low Simple Novice 

Low Simple Guru 

Low Simple Guru 

Low Complex Novice 

Low Complex Novice 

Low Complex Guru 

Low Complex Guru 

High Simple Novice 

High Simple Novice 

High Simple Guru 

High Simple Guru 

High Complex Novice 

High Complex Novice 

High Complex Guru 

High Complex Guru 

Source: Data courtesy of Dan Porter. 


Problem 

Assessment 

Tune 


Problem 

Implementation 

Time 


Total 

Resolution 

Time 
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Chapter 





Multivariate Linear 
Regression Models 




7.1 Introduction 


Regression analysis is the statistical methodology for predicting values of one or 
more response (dependent) variables from a collection of predictor (independent) 
variable values. It can also be used for assessing the effects of the predictor vana es 
on the responses. Unfortunately, the name regression , culled from the title o the-, 
first paper on the subject by F. Gallon [15], in no way reflects either the importance 

or breadth of application of this methodology. 

In this chapter, we first discuss the multiple regression model for the Predic¬ 
tion of a single response. This model is then generalized to handle the 
of several dependent variables. Our treatment must be somewhat terse, as avast 
literature exists on the subject. (If you are interested in pursuing regr 
analysis, see the following books, in ascending order ofdifficu ty. rai am a 

Ledolter [1], Bowerman and O’Connell [6], Neter, Waterman, Kutner, and 
Nachtsheim [20], Draper and Smith [13], Cook and Weisberg! [ ]- e J T 
and Goldberger [16].) Our abbreviated treatment highhghtsthe regrcs^ 
assumptions and their consequences, alternative formulations ® , jf _ 

model and the general applicability of regression techniques to seemingly dif 
ferent situations. 


7.2 The Classical Linear Regression Model 


Let z\ , Z 2 , • • be r predictor variables thought to be related to a response 
Y. For example, with r = 4, we might have 

Y = current market value of home 


variable 



and 


Z\ = square feet of living area 

Z 2 = location (indicator for zone of city) 

z 3 = appraised value last year 

z 4 = quality of construction (price per square foot) 

The classical linear regression model states that Y is composed of a mean, which de¬ 
pends in a continuous manner on the z/s, and a random error e, which accounts for 
measurement error and the effects of other variables not explicitly considered in the 
model. The values of the predictor variables recorded from the experiment or set by 
the investigator are treated as fixed. The error (and hence the response) is viewed 
as a random variable whose behavior is characterized by a set of distributional 
assumptions. 

Specifically, the linear regression model with a single response takes the form 

Y = A) + £iZi+- + /3 r z r +e 

[Response] = [mean (dependingon Zj,Z2,• • •, Z r )] + [error] 

The term “linear” refers to the fact that the mean is a linear function of the un¬ 
known parameters p 0 , Pi, - -., fi r . The predictor variables may or may not enter the 
model as first-order terms. 

• With n independent observations on Y and the associated values of z t , the com¬ 
plete model becomes 


Y\ = 00 + £lZll + 0 2 Zi2 + - * * + PrZir + Cl 

*2 = Po + @1*21 + P 2 Z 22 + * " + PrZ2r + 2 

Y n ~ P 0 + P\Z n \ + P2^n2 + + PrZnr + B „ 

where the error terms are assumed to have the following properties: 

1. E( Bj ) = 0; 

2. Var(e y ) = a 2 (constant); and 

3. Cov(e ; -,e*) — 0,y ^ k. 

In matrix notation, (7-1) becomes 


"y,~ 


1 Zll Z\ 2 *■* Z\ r 


Po 



y 2 

= 

1 Z2\ Z22 "* Z2r 


01 

+ 


Y„_ 


Zn 1 Z n 2 ' * ’ Z nr _ 


,Pr. 


^ n _ 


Y = Z p + £ 

(«X 1 ) (nX(r+l))(( r +l)xl) («Xl) 

and the specifications in (7-2) become 


(7-1) 


(7-2) 


1. E(e) = 0; and 

2. Cov(e) = E(ee') = cr 2 !. 


360 
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Note that a one in the first column of the design matrix Z is the multiplier of the 
constant term p 0 . It is customary to introduce the artificial variable zjo = 1, so that 

P 0 + PlZjl + '' * + PrZjr = PoZjO + PlZjl + ' *' + PrZji 

Each column-of Z consists of the n values of the corresponding predictor variable, 
while the yth row of Z contains the values for all predictor variables on theyth trial. 

Classical Linear Regression Model J 

y = z p + e , - 

(nXl) (nx(r+l)) ((r+l)Xl) (« X D 

E(e) = 0 and Cov (e) = <r 2 I, (7-3) 

where p and cr 2 are unknown parameters and the design matrix Z hasyth row- 
[^/0’ Zjl i • • • » Zjr ]• 

Although the error-term assumptions in (7-2) are very modest, we shall later need 
to add the assumption of joint normality for making confidence statements and 
testing hypotheses. 

We now provide some examples of the linear regression model. 

Example 7.1 (Fitting a straight-line regression model) Determine the linear regression 
model for fitting a straight line 

Mean response = E(Y) = Po + P\ z \ 

to the data 


Zi 

0 

1 

2 3 

4 

y 

1 

4 

w 

00 

9 


Before the responses Y' = [Y U Y 2 ,...,Y 5 ] are observed, the errors e' - 
[e 1 , e 2 ,..., e 5 ] are random, and we can write 

Y = ZP + e 

where 
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The data for this model are contained in the observed response vector y and the 
design matrix Z, where 



Note that we can handle a quadratic expression for the mean response by intro¬ 
ducing the term p 2 Z 2 , with z 2 = z ?. The linear regression model for the yth trial in 
this latter case is 

Y j = Po + PlZjl + p2Zj2 + e j 

or 

Yj = Po + PlZjl + p 2 zji + £j ■ 

Example 7.2 (The design matrix for one-way ANOVA as a regression model) 

Determine the design matrix if the linear regression model is applied to the one-way 
ANOVA situation in Example 6.6. 

We create so-called dummy variables to handle the three population means: 
!±i = fi + Tj, = fjL + t 2 , and fjL 3 = (jl + r 3 . We set 

! 1 if the observation is f 1 if the observation is 

from population 1 Z 2 = \ from population 2 
0 otherwise [0 otherwise 

{ 1 if the observation is 
from population 3 
0 otherwise 

and P 0 = /jL,p x = t Xj P 2 = r 2 ,p 3 = r 3 . Then 

Yj = Po + + /3 2 Z 7 -2 + /3 3 Z; 3 + £;, j = 1,2, ...,8 

where we arrange the observations from the three populations in sequence. Thus, we 
obtain the observed response vector and design matrix 

“9 "I fl 1 0 0 " 

6 110 0 

9 110 0 

Y = 0 . z = 1 0 1 0 

(8xl) 2 ’ (8x4) 10 10 

3 10 0 1 

1 10 0 1 

- 2 J Li 0 0 ij m 

The construction of dummy variables, as in Example 7.2, allows the whole of 
analysis of variance to be treated within the multiple linear regression framework. 
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7.3 Least Squares Estimation i 

One of the objectives of regression analysis is to develop an equation that will alio 
the investigator to predict the response for given values of the predictor variables* 
Thus, it is necessary to “fit” the model in (7-3) to the observed y ; corresponding! 
the known values 1, ..., Zj r - That is, we must determine the values for the? 

regression coefficients p and the error variance cr 2 consistent with the available data? 


Let b be trial values for p . Consider the difference y ; - - b 0 - biZji - 

between the observed response and the value + biZji + • ■ • + b r z ]r that woiil| 
be expected if b were the “true” parameter vector. Typically, the differenc 
- bo ~ b\Zj\ — * ■ * — b r Zjr will not be zero, because the response fluctuates (ifrf 
manner characterized by the error term assumptions) about its expected value. Th|P 
method of least squares selects b so as to minimize the sum of the squares of thl 
differences: ‘7 

5(b) = 2 (y) - bo ~ b } z n - brZjr) 2 J 

y-i (7=4jt 

= (y - Zb)'(y - Zb) ^ 

The coefficients b chosen by the least squares criterion are called least squares esti¬ 
mates of the regression parameters p. They will henceforth be denoted by p to em¬ 
phasize their role as estimates of p. 

The coefficients p are consistent, with the data in the sense that they produce* 
estimated (fitted) mean responses, p 0 + fiiZji + ••• + p r Zj r , the sum of whose- 
squares of the differences from the observed y ; is as small as possible. The deviations! 




PO ~ PlZjl - PrZjr , 


1,2 


are called residuals. The vector of residuals e = y — Z p contains the information : 
about the remaining unknown parameter a 2 . (See Result 7.2.) 

Result 7.1. Let Z have full rank r + 1 n} The least squares estimate of p in 
(7-3) is given by 

P = (Z'Zr'Z'y 

Let y = Zp = Hy denote the fitted values of y, where H = Z(Z'Z) *Z' is called 
“hat” matrix. Then the residuals 

e = y - y = [I - Z(Z'Z) _ 1 Z']y = (I - H)y 

satisfy Z'e = 0 and y'e = 0. Also, the . 


residual sum of squares = ^ (_y ; - - piZji — ••• - p r Zj r ) ~ 

= y'[I - Z(Z'Z) - 1 Z']y = y'y - y"ZP 


’If Z is not full rank, (Z'Z) 1 is replaced by (Z'Z) , a generalized inverse of Z Z. (Se 
Exercise 7.6.) 
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Proof. Let j} - (Z'Z)~‘z'y as asserted. Then e=y-y= y - Z fi = 
[I Z(Z'Z) Z']y. The matrix [I - Z(Z'Z) _1 Z'j satisfies 

1. [I Z(ZZ) Zj = [I - Z(Z'Z) 1 Z I ] (symmetric); 

2. [I - Z(Z'Z)~ 1 Z / ][I - Z(Z'Z) _1 Z'] 

= I - 2Z(Z'Z)~‘Z' + Z(Z'Z) -1 Z'Z(Z'Z) _1 Z' 0-6) 

= ~ Z(Z'Z) 1 Z'j (idempotent); 

3. Z'[I - Z(Z'Z)- 1 Z'] = Z - Z' = 0. 

Consequently, Z'e = Z'(y - y) = Z'[I - Z{Z'Z)~'Z'\y = 0, so y'e = p'Z'e = 0. 
Additionally, e'e = y'[I-Z(Z'Z)-Z'Jfl-Z^^Z'Jy = y'[I-Z(Z'Z)- 1 Z']y 
y y y verify the expression for f ), we write 

y - Zb = y - Zp + Zp - Zb = y - Zp + Z (p - b) 
so 

5(b) = (y - Zb)'(y - Zb) 

= (y - Zp)'{y - zp) + (p - b)'Z'Z(p - b) 

+ 2(y - ZP)'Z(P - b) 

= (y - zp)'(y - Zp) + Cp~ b)'Z'Z(P - b) 

fZ n ( J~ h ZiiyZ !f Z T 0,The . first term in 5(b) does not depend on b and the 
econd is the squared length of Z (p - b ). Because Z has full rank, Z (P - b) * 0 

fZ^rV S0 M th f of squares is unique and occurs for b = p = 

of fuH ra / 7 ° tC ^ Z ex,sts since ZZ has rank r + 1 s n. (If Z'Z is not 

°radtos 7 h l ^ * * °* bUt then a ' Z ' Za = 0 Za = 0. which con- 

tradicts Z having full rank r + 1.) 

obtaine^from theT h ° W SqUarCS estimates P and the residuals e can be 

obtained from the design matrix Zand responses y by simple matrix operations. 


estimates ’ th? residuals, and the residual 

residMl^ mTfln^ r h * e3St Square estimates p, the residuals e, and the 
residual sum of squares for a straight-line model 

= /3q + PlZjl + Ej 

fit to the data 


Zi 

0 

1 

2 

3 

4 

y I 

1 

4 ~ 

3 

8 

9 
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We have 


11111 
0 12 3 4 


5 10 
10 30 


.6 -.2 

-.2 .1 


Consequently, 


and the fitted equation is 


-i 1” 6 ~ 2 25 - 1 

(Z'Z) Z y - _ j 70 I |_2 


5> = 1 + 2z 


The vector of fitted (predicted) values is 


y = Z/}= 1 2 = 5 

1 3 , L J 7 


e = y - y = 3 - 5 - 2 

8 7 1 

9 |_ 9 J L °- 


The residual sum of squares is 


i'a = [0 1-2 10] -2 = 0 2 + l 2 + ( _ 2) 2 + l 2 + 0 2 = 6 


Sum-of-Squares Decomposition 

n 

According to Result 7.1, y'e = 0, so the total response sum of squares y'y = 2 
satisfies 

y'y = (y + y - y)'(y + y - y) = (y + *)'(y + 5) = y'y + < 7 


Least Squares Estimation 367 


Since the first column of Z is 1, the condition Z'e = 0 includes the requirement 

n n n __ _ 

0 = l'e = ^ Ej = 2 yj ~~ 2 h' or y = y. Subtracting riy 2 = n(y) 2 from both 

7=1 7=1 

sides of the decomposition in (7*7), we obtain the basic decomposition of the sum of 
squares about the mean: 

y'y - ny 2 = y'y - n(y) 2 + e'e 


2 O'; - yf = 2 (iv - yf + 2 ®/ 

;=l ;=i ;=i 


f f alSUm | / regression \ / residua| (error) \ 

of squares = I sum of -f- , v 

, \ / J ( sum of squares / 

y about meany y squares j \ n / 

The preceding sum of squares decomposition suggests that the quality of the models 
fit can be measured by the coefficient of determination 


R 2 = 1 - 


23 2(>;-y) 2 

;=t __ m _ 

2 (y> - y) 2 2 (yy - ?) 2 

/=! ;=1 


The quantity 7? 2 gives the proportion of the total variation in the yfs “explained” 
by, or attributable to, the predictor variables z \, Z2» * • •»Here R 2 (or the multiple 
correlation coefficient R ~ + V7?) equals 1 if the fitted equation passes through all 
the data points, so^ that e } = 0 for all j. At the other extreme, R 2 is 0 if p 0 = y and 
Pi = P 2 — ■ * ■ = P r ~ 0. In this case, the predictor variables Z\, Z 2 ,. • ■, Zr have no 
influence on the response. 

Geometry of Least Squares 

A geometrical interpretation of the least squares technique highlights the nature of 
the concept. According to the classical linear regression model, 


Mean response vector = £(Y) = Zp = p 0 


Thus, £(Y) is a linear combination of the columns of Z. As p varies, Z p spans the 
model plane of all linear combinations. Usually, the observation vector y will not lie 
in the model plane, because of the random error e ; that is, y is not (exactly) a linear 
combination of the columns of Z. Recall that 


response 

vector 


vector 
in model 
plane 



hr 



Once the observations become available, the least squares solution is derived 
from the deviation vector 

y - Zb = (observation vector) - (vectorm model plane) 

,, / ZbVfv - Zb) is the sum of squares 5(b). As illustrated in 

The squared eng h (y ^ such ^ zb is the point m 

F ' 8 ° re jpi’flane closest to y.Ttis point occurs at the tip of the perpendicular pro- 
the model plane closes y F hoice b = ft, y = Zp is the projection of 

jection of y on the Jane. Th ^ combinations of the columns of Z.The residual 

;ZTi >» ,ha ' pla "™ s hoMs ' ven " he ” Zis 

notof funraik. projection operation is expressed analytically as 

When Z has full . V I spectral decompo- 

multiplication by the matrix Z(Z Z) Z . lo see mis, we 

sition ( 2 - 16 ) to write 

Z'Z = Ajeiej + A 2 e 2 e 2 + ' " + ^+i c <-+l e >-+i 

, 1^1 a + , > 0 are the eigenvalues of Z'Z and e 1 ,e 2 ,..-,e,+i are 

where A, s A 2 * _ - **+ f . 


Aecorresponding eigln rectors. If Z is of full rank, 


1 ,1 , , 
(Z'Z) =-e,e,+~e 2 e 2 +- 


+ --e r +l e r+l 

^T + l 


/l l * 

n -A-^Ze which is a linear combination of the columns of Z.Thenqjqt 

? S T"' a“a:Wa^ = 0 if i *k or 1 if i = *. That is, the r + 1 

~ A ‘ Xk ‘ mut u a iiv perpendicular and have unit length. Their linear combina- 
lions spu] the spaoe)tfa]l linear combinations of Che ccJumns of Z. Moreover, 

z{z'z)' l z' = 2 A « lze i e ! z ' = 2 ^ 
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According to Result 2A.2 and Definition 2A. 12, the projection of y on a linear com- 

bination of {q I ,q 2 ,-.. ,q,+i} is 2 (<l/y)<l/ = (Eh 1 )! = Z(Z'Z) _1 Z'y = Zp 

Thus, multiplication by Z (Z'Z) ! Z' projects a vector onto the space spanned by the 
columns of Z. 2 

Similarly, [I - Z(Z'Z) *Z r ] is the matrix for the projection of y on the plane 
perpendicular to the plane spanned by the columns of Z. 

Sampling Properties of Classical Least Squares Estimators 

The least squares estimator p and the residuals £ have the sampling properties 
detailed in the next result. 

Result 7.2. Under the general linear regression model in (7-3), the least squares 
estimator p - (Z'Z) -1 Z'Y has 

E{p) = P and Cov(P) = ^(Z'Z) -1 
The residuals £ have the properties 

E(e) = 0 and Cov(e) = tr 2 [I - Z(Z'Z) -1 Z'J = <r 2 [I - H] 
Also,E(e’e) = (n — r — ljir 2 , so defining 

2 _ e'e __ Y'[l - Z(Z'Z)~ 1 Z']Y _ Y (1 - H]Y 

5 w — (r + 1) n ~~ r 1 n - r - 1 

we have 

£(s 2 ) = a 1 

Moreover, p and £ are uncorrelated. 

Proof. (See webpage: www.prenhali.com/statistics) ■ 

The least squares estimator p possesses a minimum variance property that was 
first established by Gauss. The following result concerns “best” estimators of linear 
parametric functions of the form c' p = cqPq + c x P\ + * • * + c r p r for any c. 

Result 7.3 (Gauss' 3 least squares theorem). Let Y = Zp + c, where E(e) ~ 0, 
Cov (e) — a 2 1, and Z has full rank r+1. For any c, the estimator 

C p = CqPq 4- dPi + - - * + CrPr 

r,+1 

2 If Z is not of full rank, we can use the generalized inverse (Z'Z)~ - 2 €,■«/, where 

i-1 

Aj > A 2 & st A, J+1 > 0 = A, I+2 = *• - A f+ i, as described in Exercise 7.6. Then Z(Z'Z) Z' 

r,+l 

~ 2 has ra °l c r i + f and generates the unique projection of y on the space spanned by the linearly 

I =*1 

independent columns of Z.This is true for any choice of the generalized inverse. (See [23].) 

3 Much later, Markov proved a less general result, which misled many writers into attaching his 
name to this theorem. 
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of c p has the smallest possible variance among all linear estimators of the form 
a , Y = a 1 y 1 + a 2 r 2 + -- + a n Y n 
that are unbiased for c'p . 


Proof. For any fixed c, let a'Y be any unbiased estimator of c'p . Theij 
£(a'Y) = c 'p, whatever the value of ft. Also, by assumption, £(a'Y) =5 
E(a'Zp + a'e) = a'Z p. Equating the two expected value expressions yield! 
a'Z p = c 'p or>(c' - a'Z)/J = 0 for all p y including the choice p = (c # - a 'Z) 
This implies that c' = a'Z for any unbiased estimator. 

Now, c'p - c'(Z'Z) _I Z'Y = a*'Y with a* = Z(Z'Z)“ 1 c. Moreover, frorg 
Result 7.2 E(p) - p,soc'p = a* 'Y is an unbiased estimator of c'p. Thus, for an 
a satisfying the unbiased requirement c' = a'Z, 

Var(a'Y) = Var(a'Z p + a'e) = Var(a'e) = a'Icr 2 a 
= o- 2 (a - a* + a*)'(a - a* + a*) 

= a 2 [(a - a*)'(a - a*) + a*'a*] 

since (a - a*)'a* = (a - a*)'Z(Z'Zy l c = 0 from the condition (a - a*)'Z = 
a'Z - a*'Z = c' - c' = O'. Because a* is fixed and (a - a*)'(a - a*) is positive 
unless a = a*, Var (a'Y) is minimized by the choice a*'Y = c'(Z'Z) _ 1 Z'Y = c'p. 




This powerful result states that substitution of p for p leads to the best estima-^ 
tor of c 'p for any c of interest. In statistical terminology, the estimator c'p is called 1 
the best (minimum-variance) linear unbiased estimator (BLUE) of c ' p. 


7.4 Inferences About the Regression Model 


We describe inferential procedures based on the classical linear regression model in 
(7*3) with the additional (tentative) assumption that the errors e have a normal dis¬ 
tribution. Methods for checking the general adequacy of the model are considered 
in Section 7.6. 


Inferences Concerning the Regression Parameters 

Before we can assess the importance of particular variables in the regression function 

E(Y) = Pq "b P\Z\ + * *' + firZr 0^) 

we must determine the sampling distributions of p and the residual sum of squares, 
e'e . To do so, we shall assume that the errors e have a normal distribution. 

Result 7.4. Let Y = Zp + e, where Z has full rank r + 1 and e is distributed as 
N n (0,<r 2 I). Then the maximum likelihood estimator of p is the same as the least 
squares estimator p . Moreover, 

P = (Z'Z) -1 Z'Y is distributed as AMfto^Z'Z) -1 ) 
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and is distributed independently of the residuals e = Y - Zp. Further, 
no- 2 = e'e is distributed as o- 2 xl- r -i 
where a is the maximum likelihood estimator of <r 2 . 

Proof. (See webpage: www.prenhall.com/statistics) m 

A confidence ellipsoid for p is easily constructed. It is expressed in terms of the 
estimated covariance matrix s 2 (Z'Z) -1 , where j 2 = e'e/(n - r - 1). 

7 * 5 * x Let Y = z P + e, where Z has full rank r + 1 and e is N n ( 0, <r 2 I). Then 
a iuo( 1 - a) percent confidence region for p is given by 

(P-P)'Z Z(P-P) s ( r + 1 )s 2 /v +ljn . r _,(a) 

where (a) is the upper (100a)th percentile of an /’-distribution with r + 1 

and n - r - 1 d.f. 

Also, simultaneous 100(1 - a) percent confidence intervals for the p. are 
given by 

Pi ± VV^(ft) V (r + ljFrZ^ZI&j, i = 0,1,...,r 
where Var(ft) is the diagonal element of s 2 (Z'Z) _1 corresponding to ft. 

Proof. Consider the symmetric square-root matrix (Z'Z) 1/2 . (See (2-22).l Set 
V = (Z'Z) (p - p) and note that E(\) = 0, 

Cov(V) = (Z'Z) 1/12 Cov (p) (Z'Z) 1/2 = CT 2 (Z'Z) I/2 (Z'Z)- ] (Z'Z) 1/2 = a 2 \ 

and V is normally distributed, since it consists of linear combinations of the ft’s. 
Therefore, V'V = (p - P)'(Z'Z)^(Z'Z) 1/2 (p - p) = ~ P)'(Z‘Z){P - p) 

is^ distributed as <r 2 * r 2 +1 . By Result 7.4 („ - r - l)s 2 = e'e is distributed as 
<* Xn-r-i, independently of p and, hence, independently of V. Consequently, 
lXt+i/(r + 1 MxZ-r-i/in - r - 1)] = [V'V/(r + l))/s 2 has an F r+l distri¬ 
bution, and the confidence ellipsoid for p follows. Projecting this ellipsoid for 
(P ~ P) using Result 5A.1 with A -1 = Z'Z/s 2 , c 2 = (r + l)F r+ ,_ l( «), and u' = 

[0^..., 0,1,0,..., 0] yields | ft - ft | s V(r + l)/v +1 , n _ j(a ) VVar(I), where 
Var (ft) is the diagonal element of j 2 (Z'Z) _1 corresponding to ft . m 

The confidence ellipsoid is centered at the maximum likelihood estimate B, 
and ns orientation and size are determined by the eigenvalues and eigenvectors of 
z Z. If an eigenvalue is nearly zero, the confidence ellipsoid will be very long in the 
direction of the corresponding eigenvector. 
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Practitioners often ignore the “simultaneous” confidence property of the inter¬ 
val estimates in Result 7.5. Instead, they replace (r + l)fv+i.„-,-](<*) with the one- - 
at-a-time t value f n - r -i(<*/2) and use the intervals 

b ± (7-ii): 

when searching for important predictor variables. 


Example 7.4 (Fitting a regression model to real-estate data) The assessment data in • 
Table 7.1 were gathered from 20 homes in a Milwaukee. Wisconsin, neighborhood. 
Fit the regression model 

Yj = p 0 + PiZji + Pi z ji + e i 

where z, = total dwelling size (in hundreds of square feet),z 2 = assessed value (in 
thousands of dollars), and Y = selling price (in thousands of dollars), to these data 
using the method of least squares. A computer calculation yields 
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and 

[ 30.967“ 

p = (Z'Z) _1 Z'y = 2.634 

.045 _ 

Thus, the fitted equation is 

y « 30.967 + 2.634^ + .045* 2 

(7.88) (.785) (.285) 

with 5 = 3.473. The numbers in parentheses are the estimated standard deviations 
of the least squares coefficients. Also, R 2 = .834, indicating that the data exhibit a 
strong regression relationship. (See Panel 7.1, which contains the regression analysis 
of these data using the SAS statistical software package.) If the residuals e pass 
the diagnostic checks described in Section 7.6, the fitted equation could be used 
to predict the selling price of another house in the neighborhood from its size 


SAS ANALYSIS FOR EXAMPLE 7.4 USING PROC REG. 


title 'Regression Analysis'; 

data estate; 

infile 'T7-1.dat'; 

input zl z2 y; 

proc reg data = estate; 

model y = zl z2; 


PROGRAM COMMANDS 


Model: MODEL 1 
Dependent Variable: 


OUTPUT 


Analysis of Variance 


Source 

DF 

Squares 

Square 

F value 

Prob > F 

Model 

2 

1032.87506 

516.43753 

42.828 

0.0001 

Error 

17 

204.99494 

12.05853 



C Total 

19 

1237.87000 





Root MSE 

Deep Mean 
C.V. 


76.55000 

4.53630 


R-square 
Adj R-sq 


Parameter Estimates 


Variable 

INTERCEP 

zl 

z2 


Parameter 

Estimate 

30.966566 

2.634400 

0,045184 


Standard 

Error 

7.88220844 

0.78559872 

0.28518271 


T for HO: 
Parameter = 0 
3.929 
3.353 
0.158 


Prob > ITI 
0.0011 
0.0038 
0.8760 
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and assessed value. We note that a 95% confidence interval for ft [see (7-14)] is 
given by 


p 2 ± r 17 (.025) V Var(ft) = .045 ± 2.110(.285) 


(-.556,-647) 

Since the confidence interval includes ft = 0, the variable z 2 might be dropped 
from the regression model and the analysis repeated with the single predictor vari¬ 
able z,. Given dwelling size, assessed value seems to add litde to the prediction of 
selling price. * 

Likelihood Ratio Tests for the Regression Parameters 

Part of regression analysis is concerned with assessing the effects of particular pre- 
dictor variables on the response variable. One null hypothesis of interes sta es that 
certain of the z,’s do not influence the response Y .These predictors will be labeled 
z 4+ 1 , z q+2 ,. . . ,z r .The statement that z q+i , z q+2 , -,z r do not influence Y translates 

into the statistical hypothesis 

H 0 '. ft + i = 0*+2 = • ■■ ■ = Pr = 0 or H 0 : fi m = • (7-12) 

where jS( 2 ) = [ft+i>ft+ 2 > 

Setting 

r 1 1 r p w 

! - a- ifetHSIi 

z = Z! ; Z 2 , P - Bm 

L«x(, + 1) ; «X(^*)J L(fr-f)«)J 

we can express the general linear model as 


Y = Zfi + e = [Z] I Z 2 ] ~ !?) + e = Z i/*(U + Z 20(2) + e 


Under the null hypothesis H 0 : fi m = 0, Y = Zjft,) + likelihood ratio test 

of Ho is based on the 

Extra sum of squares = SS res (Zi) - SS re5 (Z) ^ ^ 

= (y - z ifti))'(y - Z i/8(1)) - (y - zp y(y - 
where ft,) = (ZJZi) Z[y. 

Result 7 6 Let Z have full rank r + 1 and e be distributed as A«0,cr 2 I). The 
SL^aS test of = • «• equivalent to a test of H 0 based on the 

extra sum of squares in (7-13) and i 2 = (y - W) (y " Z P)K n r In 
particular, the likelihood ratio test rejects Ho if 

(SS res (Zi) - SS res ( z ))/(f - g) > j? r _^ n _ r _ t (a) 

S 2 

where F r . q ,„- r -,(«) is the upper (10te)th percentile of an /--distribution with r - q 
and n - r — 1 d.f. 
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Proof. Given the data and the normal assumption, the likelihood associated with 
the parameters p and <t 2 is 

L(py) = -L_ e -(y-zp)'(y-zp)/ 2 a 2 s -1—^/2 

( ’ (2 ir)"'V (2ir) n/2 ? 1 

with the maximum occurring at p = (Z'Z) - 1 Z f y and o 2 = (y - Z/?)'(y - Z p)/n. 
Under the restriction of the null hypothesis, Y = Z \p^ > + e and 

max L(pn\ y cr 2 ) = 1 n - e~ nl1 

{) ’ (2ir)^ 

where the maximum occurs at p^ = (ZiZ^ ^Zly. Moreover, 

°\ = (y - z iJ8(i))'(y - ZjjSf!) )/n 
Rejecting H 0 : p( 2 ) = 0 for small values of the likelihood ratio 


max /.(ftp,cr 2 ) /~2 

max L(p, a 2 ) \a 2 

P,cr 2 


YY n/2 


(J 2 + cr\ — or 


= i + 


5? - & 2 Y n/2 


is equivalent to rejecting H 0 for large values of (<xj - 6 2 )/d- 2 or its scaled version, 

n{% ~ PW - q) _ (SS res (Zt) - SS res (Z))/(r - q) 
no^Kn — r — 1) s 2 

The preceding F’-ratio has an F’-distribution with r - q and n - r - 1 d.f. (See [22] 
or Result 7.11 with m — 1.) ■ 


Comment . The likelihood ratio test is implemented as follows. To test whether 
all coefficients in a subset are zero, fit the model with and without the terms corre¬ 
sponding to these coefficients.The improvement in the residual sum of squares (the 
extra sum of squares) is compared to the residual sum of squares for the full model 
via the F’-ratio. The same procedure applies even in analysis of variance situations 
where Z is not of full rank. 4 

More generally, it is possible to formulate null hypotheses concerning r — q lin¬ 
ear combinations of p of the form H 0 : Cp = A 0 .Letthe(r - q) X (r + 1) matrix 
C have full rank, let A 0 = 0, and consider 

H 0 : Cp = 0 


^This null hypothesis reduces to the previous choice when C = 0 


(r-q)X(r-q) 


4 In situations where Z is not of fuJJ rank, rank(Z) replaces r + 1 and rank(Z t ) replaces q + 1 in 
Result 7.6. 
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Under the full model, C/3 is distributed as N,_,(C/),< 7 2 C(Z'Z) ! C'). We reject* 
Ho'. C/3 = 0 at level a if 0 does not lie in the 100(1 - a)% confidence ellipsoid fori 
C/8. Equivalently, we reject Hq- Cp = 0 if 


(C/3)'(C(Z'Z )- 


> (r ~ q)Fr- q 


Zp)'(y ~ ZP)/(n - r - 1) and F,- tlll - f -iW is the upper 


W11C1C J \J ' *• V ' ' ” 1 , r ry. * . ^ 

(100a)th percentile of an /’-distribution with r - q and n - r - l d.L Uie test m3 
(7-14) is the likelihood ratio test, and the numerator in the E-ratio is the extra residua^ 
sum of squares incurred by fitting the model, subject to the restriction that Cp = 0. 

(See [23]). . \ 1 

The next example illustrates how unbalanced experimental designs are easily 

handled by the general theory just described. 




Example 7.5 (Testing the importance of additional predictors using the extra sum-of-J 
squares approach) Male and female patrons rated the service in three establish¬ 
ments (locations) of a large restaurant chain. The service ratings were converted 
into an index. Table 7.2 contains the data for n = 18 customers. Each data point in 
the table is categorized according to location (1,2, or 3) and gender (male = 0 and 
female = 1). This categorization has the format of a two-way table with unequal 
numbers of observations per cell. For instance, the combination of location 1 and 
male has 5 responses, while the combination of location 2 and female has 2 respons¬ 
es. Introducing three dummy variables to account for location and two dummy vari¬ 
ables to account for gender, we can develop a regression model linking the service 
index Y to location, gender, and their “interaction” using the design matrix 




Tabic 7.2 Restaurant-Service Data 


Location 


Gender 


Service (K) 
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location gender interaction 

100 10 10000 oV 

100 1 0 100000 

100 10 100000 >5 responses 

100 10 100000 

100 10 100000 , 

100 01 0 1 0 0 0 0 1 

1 0 0 0 1 0 1 0 0 0 0 } 2res P° nses 

010 10 001000 

010 10 001000 

010 10 001000 

010 10 001000 

010 10 001000 

010 01 0 0 0 1 0 0 1 

0 1 0 0 1 0 0 0 1 0 0 } 2res P° nses 

001 10 0 0 0 0 1 0 1 

0 0 1 1 0 0 0 0 0 1 0 } 2res P° nses 

001 01 000001 

0 0 1 0 1 0 0 0 0 0 1 . J J 2 res P°n ses 

The coefficient vector can be set out as 

P r = [£o,0l,02*03»Ti,T 2 ,y n ,y 12l 721,722,7317732] 

where the /Vs (i > 0) represent the effects of the locations on the determination of 
service, the r/s represent the effects of gender on the service index, and the y/*’s 
represent the location-gender interaction effects. 

The design matrix Z is not of full rank. (For instance, column 1 equals the sum 
of columns 2-4 or columns 5-6.) In fact, rank(Z) = 6. 

For the complete model, results from a computer program give 

SS res (Z) = 2977.4 

and n - rank(Z) = 18 — 6 = 12. 

Hie model without the interaction terms has the design matrix Z 2 consisting of 
the first six columns of Z. We find that 

SS res (Z!) = 3419.1 

with n - rank(Zj) = 18 - 4 = 14.To test H 0 : y 11 = y 12 = y 2 i = 722 = 731 = 

732 = 0 ( no location-gender interaction), we compute 

(SS^ZQ - SS res (Z))/(6 - 4) = (SS res (Z 2 ) - SS res (Z))/2 
5 2 SS res (Z)/12 

(3419.1 - 2977.4)/2 

2977.4/12 ~ ' 89 



constant 
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Th. F ratio mav be compared with an appropriate percentage point of an - 
The ^ratio may P . f ^ F _ ratio is not significant for any reasonable sig- 

F-distribution with 2 and ^ dTTh.s ^ra » , nde x does not depend 

mficance level a. Consequentsly we con,^ ^ can ^ dropped {rom ^ 

upon any location-gender interaction, a 

modeI ' „ r orxorrtarh we mav verify that there is no differ- 

Using the extra sum-of-squares approach, y y significant; that is, 

ence between locations (no location effect), but mat g 8 

males and females do not give the ^ cc || mun t s are unequal, the varia- 

In analysis-of-variance si ua 10 predictor variables and their interac- 

li'hou. lh, term, in ,nd conrprr« lire appropria* 

F-test statistics. 

7.5 Inferences from the Estimated Regression Function 

One* a, investigator is satisfiedIwiih the M 

solve two prediction problems .L?t z 0 [ ■_® 1 ’ , J ti te the regression func- 

^^-vaineortSeresponser 

at z 0 . 

Estimating the Regression Function at z 0 

, i f rpenonse when the predictor variables have values 

* <»>.*• — - y »' s 

£(y 0 lz 0 ) = /So + Pizoi + •" + P' z ° r = z ' oti (745) 

Its least squares estimate is z' 0 p . 

Result 7.1. For rhe line,, regression model in (7-3). zW> * £ 

I" S 

E(Y n \z 0 ) = z'o P is provided by 


z'oP i tn-r- 


/(zo(rzy > zo)s 2 


where Wl (a/2) is the upper 100(a/2)th percentile of a '-distribution with 
« - r - 1 d.£ 

„ r i-t r A n W \ust a linear combination of the ft’s, so Result 

Proof. For a fixed z 0 ,z 0 /M s just a nnc 7 , yr i 2 since Cov(fl) 

uted. Result 7.4 asserts that p is N r+ APi°- ( z > maepenaeni y / 





Inferences from the Estimated Regression Function 379 


is distributed as Xn-r-i/(n ~ r ~ 1)- Consequently, the linear combination z§p is 
A'(zo/?,cr 2 zo(Z'Z)' 1 z 0 ) and 

(zqP - z' 0 p)/\/o^Z/Z)~ l z 0 _ (zqP - Zp P) 

V^wzzy\j 

is distributed as t n _ r ^ x . The confidence interval follows. ■ 


Forecasting a New Observation at z 0 

Prediction of a new observation, such asF 0 > atz o = [ 1, Zo 1 > - - • , Zor] is more uncertain 
than estimating the expected value of Y 0 . According to the regression model of (7-3), 

*0 = z'oP + e 0 
or 

(new response y 0 ) = (expected value of Y 0 at z 0 ) + (new error) 

where e 0 is distributed as iV(0, cr 2 ) and is independent of e and, hence, of p and s 2 
The errors e influence the estimators p and s 2 through the responses Y, but e 0 does not 

Result 7.8. Given the linear regression model of (7-3), a new observation Y () has 
the unbiased predictor 

*0 P = Po + PlZoi + ■ ■■ + Prior 
The variance of the forecast error Y 0 - z' 0 p is 

Var (y 0 - z’op) = o- 2 (l + z6(Z'Z) _, z 0 ) 

When the errors ehave a normal distribution, a 100(1 - a)% prediction interval for 
Y 0 is given by 


ZqP ± t n - r - 


V(1 + zq(Z'Z) ’zo) 


where r /l _ r _ 1 (a/2) is the upper 100(a/2)th percentile of a /-distribution with 
n — r — 1 degrees of freedom. 

Proof. We forecast Y 0 by z' 0 p , which estimates E{Y 0 I z 0 ). By Result 7.7, Zq/ 3 has 
E(z' 0 p) = zq/ 3 and Var(z6/3) = Zo(Z'Z)~ , z 0 cr 2 . The forecast error is then 
Y 0 - zq P = zoP + e 0 - zqP = £0 + *b(P-p)- Thus, E{Y 0 - ZqP) = F(e 0 ) + 
E(zq(P - p)) = 0 so the predictor is unbiased. Since e 0 and /? are independent, 
Var(Y 0 -Zoft) = Var(e 0 ) + Var(zo^) =a' 2 + zo(Z , Z)' 1 z 0 cr 2 = cr 2 (l 4- Zo(Z'Z) _1 z 0 ). 
If it is further assumed that e has a normal distribution, then p is 
normally distri buted, a nd so is the linear combination Y 0 - z' 0 p. Consequently, 
(Vo ~ Z o£)/Vo- 2 (1 + zo(Z'Z)~ J z 0 ) is distributed as /V(0,1). Dividing this ratio by 
V s 2 /o 2 , which is distributed as \Zx*- r -\l(n — r - 1), we obtain 

(T 0 - z'qP) 

V?0 + Zo(Z'Z) _1 z 0 ) 

which is distributed as . The prediction interval follows immediately. ■ 
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The prediction interval for 7 0 is wider than the confidence interval for estimating 
the value of the regression function E(Y 0 1 z 0 ) = z'oP The additional uncertainty in 
forecasting Y 0 , which is represented by the extra term s 2 in the expression 
^(l -f- zo(Z'Z) -1 z 0 ), comes from the presence of the unknown error term e 0 . 

Example 7.6 (Interval estimates for a mean response and a future response) Companies 
considering the purchase of a computer must first assess their future needs in order 
to determine the proper equipment. A computer scientist collected data from seven 
similar company sites so that a forecast equation of computer-hardware requirements 
for inventory management could be developed.The data are given in Thble 7.3 for 

h = customer orders (in thousands) 
z 2 = add-delete item count (in thousands) 

Y = CPU (central processing unit) time (in hours) 

Construct a 95% confidence interval for the mean CPU time, E(Y$ |z 0 ) = 
j 3 0 + piZoi + 02*02 at z o = [1,130,7.5]. Also, find a 95% prediction interval for a 
new facility’s CPU requirement corresponding to the same z 0 . 

A computer program provides the estimated regression function 

y = 8.42 + 1.08 Z\ + .42z 2 

~ 8.17969 

(Z'Z)" 1 = -.06411 .00052 

.08831 -.00107 .01440 _ 

and s = 1204. Consequently, 

z' 0 p = 8.42 + 1.08(130) + .42(7.5) = 151.97 

and s\/z'o(Z'Zy i zo = 1.204(^8928) = .71. We have r 4 (.025) = 2.776, so the 95% 
confidence interval for the mean CPU time at z 0 is 

Zo/} ± t 4 (.02S)sVzttZ’Z)~\ = 151.97 ± 2.776(.71) 

or (150.00,153.94). 



Source: Data taken from H. P. Artis, Forecasting Computer Requirements: A 
Forecaster’s Dilemma (Piscataway, NJ: Bell Laboratories, 1979). 
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Since s\/l + Zo(Z'Z) l z Q = (1.204)(1.16071) = 1.40, a 95% prediction inter¬ 
val for the CPU time at a new facility with conditions z 0 is 

z'oP ± / 4 (.025)jVi + iJfZ'Zf'zo = 151.97 ± 2.776(1.40) 

or (148.08,155.86). ■ 


.6 Model Checking and Other Aspects of Regression 

Does the Model Fit? 

Assuming that the model is “correct,” we have used the estimated regression 
function to make inferences. Of course, it is imperative to examine the adequacy of 
the model before the estimated function becomes a permanent part of the decision¬ 
making apparatus. 

All the sample information on lack of fit is contained in the residuals 

®i = Ti ~~ “ 0i*i i — • • • - (3rZlr 

£2 = Yl - 00 - 01*21-0r*2r 

e /7 T/7 00 01*nl • ■ • — p r Znr 

or 

e = [I-Z(Z'Zr 1 Z']y = [I-H]y (7-16) 

If the model is valid, each residual is an estimate of the error e ; , which is assumed to 
be a normal random variable with mean zero and variance cr 2 . Although the residuals 
e have expected value 0, their covariance matrix 0^(1 - Z(Z'Z)~ 1 Z / ] = cr 2 [I - H] 
is not diagonal. Residuals have unequal variances and nonzero correlations. Fortu¬ 
nately, the correlations are often small and the variances are nearly equal. 

Because the residuals e have covariance matrix cr 2 [I — H], the variances of the 
ej can vary greatly if the diagonal elements of H, the leverages hjj, are substantially 
different. Consequently, many statisticians prefer graphical diagnostics based on stu- 
dentized residuals. Using the residual mean square s 2 as an estimate of a 2 , we have 

Var(e ; ) = j?(1 - h fj ), j = 1,2(7-17) 

and the studenrized residuals are 



We expect the studentized residuals to look, approximately, like independent drawings 
from an 7V(0,1) distribution. Some software packages go one step further and 
studentize e } using the delete-one estimated variance s 2 (j ), which is the residual 
mean square when the jth observation is dropped from the analysis. 
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Residuals should be plotted in various ways to detect possible anomalies. For 
general diagnostic purposes, the following are useful graphs: 

1. Plot the residuals sj against the predicted values y } = p 0 + PiZji + * * * + p rZ - 
Departures from the assumptions of the model are typically indicated by two 
types of phenomena: 

(a) A dependence of the residuals on the predicted value . This is illustrated in 
Figure 7.2(a). The numerical calculations are incorrect, or a /3 0 term has 
been omitted from the model. 

(b) The variance is not constant . The pattern of residuals may be funnel 
shaped, as in Figure 7.2(b), so that there is large variability for large y and 
small variability for small y . If this is the case, the variance of the error is 
not constant, and transformations or a weighted least squares approach (or 
both) are required, (See Exercise 7.3.) In Figure 7.2(d), the residuals form a 
horizontal band. This is ideal and indicates equal variances and no depen¬ 
dence on y. 

2. Plot the residuals ej against a predictor variable, such as Z\ , or products of predic¬ 
tor variables\ such as zj or z\Z 2 - A systematic pattern in these plots suggests the 
need for more terms in the model. This situation is illustrated in Figure 7.2(c). 

3. Q-Q plots and histograms . Do the errors appear to be normally distributed? To 
answer this question, the residuals e 7 or e* can be examined using the techniques 
discussed in Section 4.6. The Q-Q plots, histograms, and dot diagrams help to 
detect the presence of unusual observations or severe departures from normal¬ 
ity that may require special attention in the analysis. If n is large, minor depar¬ 
tures from normality will not greatly affect inferences about p. 
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4. Plot the residuals versus time. The assumption of independence is crucial, but 
hard to check. If the data are naturally chronological, a plot of the residuals ver¬ 
sus time may reveal a systematic pattern. (A plot of the positions of the residu¬ 
als in space may also reveal associations among the errors.) For instance, 
residuals that increase over time indicate a strong positive dependence. A statis¬ 
tical test of independence can be constructed from the first autocorrelation, 


n 


2 ®y*y-i 

i~2 



7=1 


(7-19) 


of residuals from adjacent periods. A popular test based on the statistic 
2 (®; ~ ®;-i) j 2 53 2(1 ~ ^i) is called the Durbin-Watson test . (See [14] 

for a description of this test and tables of critical values.) 


Example 7,7 (Residual plots) Three residual plots for the computer data discussed 
in Example 7.6 are shown in Figure 7.3. The sample size n = 7 is really too small to 
allow definitive judgments; however, it appears as if the regression assumptions are 
tenable. r 



(c) 

Figure 7.3 Residual plots for the computer data of Example 7.6. 
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If several observations of the response are available for the same values of the 
predictor variables, then a formal test for lack of fit can be carried out. (See [13] f Qr 
a discussion of the pure-error lack-of-fit test.) 

Leverage and Influence 

Although a residual analysis is useful in assessing the fit of a model, departures from 
the regression model are often hidden by the fitting process. For example, there may 
be “outliers’’in either the response or explanatory variables that can have a consid¬ 
erable effect on the analysis yet are not easily detected from an examination of 
residual plots. In fact, these outliers may determine the fit. 

The leverage hy the (/', j) diagonal element of H = Z(Z'Z)~ 1 Z, can be interpret¬ 
ed in two related ways. First, the leverage is associated with the ;th data point mea¬ 
sures, in the space of the explanatory variables, how far the y'th observation is from the 
other n - 1 observations. For simple linear regression with one explanatory variable z, 

1 (*/ - z? 


The average leverage is (r + 1 )/n. (See Exercise 7.8.) 

Second, the leverage hy, is a measure of pull that a single case exerts on the fit. 
The vector of predicted values is 

y = Z0 = Z(Z Zr'Zy = Hy 

where the y'th row expresses the fitted value 1) in terms of the observations as 

y t = hjjyj + ^h jk y k 
k*j 

Provided that all other y values are held fixed 

(change in yf) = hy (change in yf) 

If the leverage is large relative to the other hj k , then y } will be a major contributor to 
the predicted value y } . 

Observations that significantly affect inferences drawn from the data are said to 
be influential. Methods for assessing influence are typically based on the change in 
the vector of parameter estimates, /?, when observations are deleted. Plots based 
upon leverage and influence statistics and their use in diagnostic checking of regres¬ 
sion models are described in [3], [5], and [10]. These references are recommended 
for anyone involved in an analysis of regression models. 

If, after the diagnostic checks, no serious violations of the assumptions are de¬ 
tected, we can make inferences about p and the future Y values with some assur¬ 
ance that we will not be misled. 


Additional Problems in Linear Regression 

We shall briefly discuss several important aspects of regression that deserve and receive 
extensive treatments in texts devoted to regression analysis. (See [10], [11], [13], and [23].) 
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Selecting predictor variables from a large set. In practice, it is often difficult to for¬ 
mulate an appropriate regression function immediately. Which predictor variables 
should be included? What form should the regression function take? 

When the list of possible predictor variables is very large, not all of the variables 
can be included in the regression function. Techniques and computer programs de¬ 
signed to select the “best” subset of predictors are now readily available. The good 

ones try all subsets: Z\ alone, z 2 alone,..., z\ and Z 2 ,_The best choice is decided by 

examining some criterion quantity like R 2 . [See (7-9).] However, R 2 always increases 
with the inclusion of additional predictor variables. Although this problem can be 
circumvented by using the adjusted R 2 , R 2 = 1 - (1 - R 2 )(n - 1 )/(n - r - 1), a 
better statistic for selecting variables seems to be Mallow’s C p statistic (see [12]), 

( residual sum of squares for subset model 
with p parameters, including an intercept 
(residual variance for full model) 

A plot of the pairs (/?, C p ), one for each subset of predictors, will indicate models 
that forecast the observed responses well. Good models typically have (p, C p ) coor¬ 
dinates near the 45° line. In Figure 7.4, we have circled the point corresponding to 
the “best” subset of predictor variables. 

If the list of predictor variables is very long, cost considerations limit the number 
of models that can be examined. Another approach, called stepwise regression (see 
[13]), attempts to select important predictors without considering all the possibilities. 




Figure 7.4 C p plot for computer 
data from Example 7.6 with 
three predictor variables 
(zj = orders, Z 2 = add-delete 
count, z 3 = number of items; see 
the example and original source). 
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Hie procedure can be described by listing the basic steps (algorithm) involved in the 
computations: 

Step 1. All possible simple linear regressions are considered. The predictor variable 
that explains the largest significant proportion of the variation in Y (the variable, 
that has the largest correlation with the response) is the first variable to enter the re¬ 
gression function. 

Step Z The next variable to enter is the one (out of those not yet included) that 
makes the largest significant contribution to the regression sum of squares. The sig¬ 
nificance of the contribution is determined by an F-test. (See Result 7.6.) The value 
of the F-statistic that must be exceeded before the contribution of a variable is 
deemed significant is often called the F to enter. 

Step 3. Once an additional variable has been included in the equation, the individ¬ 
ual contributions to the regression sum of squares of the other variables already in 
the equation are checked for significance using F-tests. If the F-statistic is less than 
the one (called the Fto remove) corresponding to a prescribed significance level, the 
variable is deleted from the regression function. 

Step 4. Steps 2 and 3 are repeated until all possible additions are nonsignificant and 
all possible deletions are significant. At this point the selection stops. 

Because of the step-by-step procedure, there is no guarantee that this approach 
will select, for example, the best three variables for prediction. A second drawback is 
that the (automatic) selection methods are not capable of indicating when transfor¬ 
mations of variables are useful. 

Another popular criterion for selecting an appropriate model, called an infor¬ 
mation criterion, also balances the size of the residual sum of squares with the num¬ 
ber of parameters in the model. 

Akaike’s information criterion (AIC) is 

/ residual sum of squares for subset model \ 

with p parameters, including an intercept 

AIC = *ln -—--- —1+2 p 

\ n ) 

It is desirable that residual sum of squares be small, but the second term penal¬ 
izes for too many parameters. Overall, we want to select models from those having 
the smaller values of AIC. 

Colinearity. If Z is not of full rank, some linear combination, such as Za, must equal 
0. In this situation, the columns are said to be colinear. This implies that Z'Z does 
not have an inverse. For most regression analyses, it is unlikely that Za = 0 exactly. 
Yet, if linear combinations of the columns of Z exist that are nearly 0, the calculation 
of (Z'Z)” 1 is numerically unstable. Typically, the diagonal entries of (Z'Z) will 
be large. This yields large estimated variances for the jB/s and it is then difficult 
to detect the “significant” regression coefficients p t . The problems caused by colin¬ 
earity can be overcome somewhat by (1) deleting one of a pair of predictor variables 
that are strongly correlated or (2) relating the response Y to the principal compo¬ 
nents of the predictor variables—that is, the rows z) of Z are treated as a sample, and 
the first few principal components are calculated as is subsequently described in 
Section 8.3. The response Y is then regressed on these new predictor variables. 
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Bias caused by a misspecified model. Suppose some important predictor variables 
are omitted from the proposed regression model. That is, suppose the true model 
has Z = [Zi \ Z 2 \ with rank r + 1 and 


Y 

("XI) 


Zi 

(nX(q-hl)) 


z 2 

(nX(r-g))] 


Pm 

(( 9 +l)xi) 

P( 2 ) 


+ £ 

(«X 1) 


— + Z 2 /J( 2) + £ 


(7-20) 


where E(e) = 0 and Var(e) = cr 2 1. However, the investigator unknowingly fits 
a model using only the first q predictors by minimizing the error sum of 
squares (Y - Z 1 ^ (1 )) , (Y - Z 1 p^). The least squares estimator of P^) is P^) = 
(ZiZO ZJY. Then, unlike the situation when the model is correct, 

E(P(i)) ~ (ZiZj) ZjF(Y) = (ZJZj) Zj(Zi/J(i) + Z 2 /3(2) + F(e)) 

= /J ( ,) + (Z\Z x )~ X Z\Z 2 p {2) (7-21) 

That is, P(i) is a biased estimator of p ^ unless the columns of Z\ are perpendicular 
to those of Z 2 (that is, Z\Z 2 = 0)^ If important variables are missing from the 
model, the least squares estimates p^ may be misleading. 


.7 Multivariate Multiple Regression 

In this section, we consider the problem of modeling the relationship between 
m responses Y\ , Y 2 , ..., Y m and a single set of predictor variables Z\ , z 2 ,..., z r . Bach 
response is assumed to follow its own regression model, so that 

Y 1 = Poi + P\\Z\ + '*• + Pr\Z r + Cl 

*2 = Po 2 + Pl2Z\ + •** + Pr2Z r + «2 (7-22) 

Y m ~ Pom + PlmZl + *** + PrmZ r + 

The error term e' = [ei, e 2 ,..., e m ] has F(e) = 0 and Var(e) = X. Thus, the error 

terms associated with different responses may be correlated. 

To establish notation conforming to the classical linear regression model, let 
[Zjo> Zji ,..., Zj r ] denote the values of the predictor variables for the yth trial, 
let Y) = [Yj i, Y /2 ,..., Y jm \ be the responses, and let e) = [e n ,e /2 ,..., e jm ] be the 
errors. In matrix notation, the design matrix 

*10 Zn ■** Zlr 

^ = *20 *21 * ' * Z2r 

(nx(r+i)) : : \ : 

-Zn 0 Z n i *** Zfif 
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is the same as that for the single-response regression model. [See (7-3).] The other 
matrix quantities have multivariate counterparts. Set 


. Y = 

(nXm) 


P 

((r+l)Xm) 


e = 

(nXm) 


I'll 

Yll 

Yu 

Y 22 

r.i 

Y„ 2 

0oi 

Pol 

0n 

P\2 

Pri 

Pr2 

e ll 

e 12 

621 

®22 

, fi nl 

2 

-tl 



=[Y„, I Y m | - | Y W ] 


[0(1) i P( 2) I ••• I 0(m)] 


E (l) I e (2) i 


The multivariate linear regression model is 


Y = z p + e 

(nXm) (nX(/-+l)) ((H-ljXm) (nXm) 


E(£ {i] ) = 0 and Cov(e (/) ,e (t) ) = tr ik I i,£ = 1,2,...,m 

The ra observations on the ;th trial have covariance matrix X = but ob¬ 
servations from different trials are uncorrelated. Here p and a ik are unknown 
parameters; the design matrix Z has ;th row [z y0 , z yl ,..., z y >]. 


Simply stated, the /th response Y(,) follows the linear regression model 

Y (l ) = Z/3 a) + e (i) , / = 1,2,...,m (7-24) 

with Cov However, the errors for different responses on the iomf trial 

can be correlated. 

Given the outcomes Y and the values of the predictor variables Z with full 
column rank, we determine the least squares estimates exclusively from the. 
observations Y^ on the /th response. In conformity with the single-response 
solution, we take 

0(0 = (Z'Zf'Z'Yfl) (7-251 
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Collecting these univariate least squares estimates, we obtain 

P = [0(1) ! 0(2) I ••• I 0(m)] = (Z'Zr'Z'fYd) i Y(2) I ••• ! Y( m) ] 

or 

fi = (Z'Z) _1 Z'Y (7-26) 

For any choice of parameters B = [b(D | b (2 ) | ••• j b(,„)], the matrix of errors 
is Y - ZB. The error sum of squares and cross products matrix is 

(Y - ZB)'(Y - ZB) 

(Y (1) - Zb ( 1 ) r(Y (1) - Zb (1) ) ••• (Y (1) - Zb ( i))'(Y (m) - Zb (m) ) 

(Y (m) - Zb (m ,)'(Y (1) - Zb (I) ) ••• (Y (m) — Zb (m) )'(Y (m) — Zb ( „,,) 

(7-27) 

The selection b(,j = f/j minimizes the ith diagonal sum of squares 
(Y(j) - Zb (0 )'(Y w - Zb (0 ). Consequently,tr[(Y - ZB)'(Y - ZB)] is minimized 
by the choice B = 0. Also, the generalized variance | (Y - ZB)'(Y - ZB) | is min¬ 
imized by the least squares estimates /J. (See Exercise 7.11 for an additional general¬ 
ized sum of squares property.) 

Using the least squares estimates fi , we can form the matrices of 
Predicted values: Y = Z/3 = Z(Z'Z)’ 1 Z'Y 

Residuals: £ = Y- Y = [I- Z(Z'Z) _1 Z']Y (7-28) 

The orthogonality conditions among the residuals, predicted values, and columns of Z, 
which hold in classical linear regression, hold in multivariate multiple regression. 
They follow from Z'[I - Z(Z'Zf ‘z'J = Z' - Z' = 0. Specifically, 

z e = Z’[I - Z(Z'Zr'Z']Y = 0 (7-29) 

so the residuals are perpendicular to the columns of Z. Also, 

Y'e = /3'Z'[I - Z(Z'Zr , Z'] Y = 0 (7-30) 



confirming that the predicted values Yp) are perpendicular to all residual vectors 
ept). Because Y = Y + £, 

Y Y = (Y + e)'(Y + e) = YY + ee + o + 0 ' 


or 

YY = Y Y 


( total sum of squares A __ /predicted sum of squares^ 
and cross products ) \ and cross products ) 


ee 


^residual (error) sum 
of squares and 
y cross products 


(7-31) 
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The residual sum of squares and cross products can also be written as 

s'e = Y'Y - Y'Y = Y'Y - ft'Z'Zfl (7 . 32) 

Example 7.8 < Fitting a multivariate straight-line regression model) To illustrate the 

A A A 

calculations of fi , Y, and £, we fit a straight-line regression model (see Panel 7.2), 

Y jl ^01 ” 1 " ” 1 " 

Y jl = /3o 2 + ^12^*1 + e ;2» / = 1,2,..., 5 

to two responses Y 1 and Y 2 using the data in Example 7.3. These data, augmented by 
observations on an additional response, are as follows: 


Zl o 

1 

2 

3 

4 

yi i 

4 

3 

8 

9 

>■2 -i 

-1 

2 

3 

2 


The design matrix Z remains unchanged from the single-response problem. We find that 


11111 
0 12 3 4 


(Z'Z)- 1 = 


.6 -.2 

-.2 .1 


PANEL 7.2 SAS ANALYSIS FOR EXAMPLE 7.8 USING PROC. GLM. 


title 'Multivariate Regression Analysis'; 
data mra; 

infile 'Example 7-8 data; 
input yl y2 zl; 
proc glm data = mra; 
model y 1 y2 = z 1/ss3; 
manova h = zl/printe; 
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PANEL 7.2 (continued) 


Type 111 SS 
40.00000000 


Mean Square 
40.00000000' 



T for HO: 
Parameter = 0 
0.91 
4.47 


Std Error of 
Estimate 
1.09544512 
0.44721360 


Source 

Model 

Error 

Corrected Total 


Sum of Squares 
10.00000000 
4.00000000 
14.00000000 


Mean Square 
10.00000000 
1.33333333 


R-Square 

0.714286 


C.V. 

115.4701 


Root MSE 
1.154701 


Y2 Mean 
1.00000000 


Type III SS 
10.00000000 


Mean Square 
10.00000000 


( Parameter 
INTERCEPT 
21 


Estimate 

- 1.000000000 

1.000000000 


T for HO: 
Parameter = 0 
- 1.12 
2.74 


Std Error of 
Estimate 
0.89442719 
0.36514837 


E = Error SS & CP Matrix 


Manova Test Criteria and Exact F Statistics for 
the Hypothesis of no Overall Zl Effect 


H = Type III SS&CP Matrix for Z1 


Statistic 

S = 1 

Value 

M =0 

F 

Wilks' Lambda 

0.06250000 

15.0000 

Pillai's Trace 

0.93750000 

15.0000 

Hotelling-Lawley Trace 

15.00000000 

15.0000 

Roy's Greatest Root 

15.00000000 

15.0000 


E = Error SS&CP Matrix 
N = 0 

Num DF D 

2 
2 
2 
2 
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Dividing each entry e\ i} e ik) of e'e by n - r - 1, we obtain the unbiased estimator 
of X. Finally, 

Cov(/*((),«(*)) = SKZ'Z^Z'eweMI - Z(Z’ZY l Z’)] 

= (Z'Z)' 1 Z'£(e w c'(* ) )(I - Z(Z'Z) _1 Z') - 
= (Z'Z) _1 ZV lt I(I - Z(Z'Z) _1 Z') 

= <r ik ((rz)~'z' - (Z'Z) _1 Z') = o 

so each element of (i is uncorrelated with each element of E. ■ 

The mean vectors and covariance matrices determined in Result 7.9 enable us 
to obtain the sampling properties of the least squares predictors. 

We first consider the problem of estimating the mean vector when the predictor 
variables have the values zj = [l,Zoi»--•»*()*■]• Th e mean res P°nse 

variable is z.'oP(;)> and this is estimated by zojB (/ ), the ith component of the fitted 
regression relationship. Collectively, 

z oP = [ z oP(i) i z oP{2) : 1 z o P(m)\ (7-34) 

is an unbiased estimator z' 0 p since £(zo/*(/)) = zo£(0(;)) = z'oP(i) f° r eac ^ compo- 
nent.From the covariance matrix for p^ and jB(*>, the estimation errors 2oP(0 “ z oP(i) 
have covariances 

£[Zo(j8(i) - P(i))(P{k) ~ P{k)) z o] = z b{E(P(i) ~ P(i))(P(k) ~ P{k)) ) Z 0 

= a, k z' 0 (Z'Zfz 0 (7-35) 

The related problem is that of forecasting a new observation vector Yb = 
[lbi.^o 2 .-->^Om] atz oAccordingtotheregressionmodel, Yoi = z 'oP(i) + Cot where 
the “new” error ej = [eoi, £ c> 2 > ■ • • > £ 0 m] is independent of the errors £ and satisfies 
E(e oi ) = Oand£(eoieot) = . The forecast error for the ith component of Y 0 is 

Yoi - iffiu) = Y 0i ~ t’oP(i) + z 'oP(i) ~ *'oh) 

= £ 0i - z o(P(i) - P( i)) 

so E(Y 0i ~ z oP(,)) = E ( £ oi) - z o E(P {i ) - Pa)) = 0, indicating that z' 0 p (i) is an 
unbiased predictor of Y 0/ . The forecast errors have covariances 

E(Y 0i - z b/8(,))(ior - z o P(k)) 

= E(e ai - z' 0 (P(i) - /3 (i )))(eo* - z 'o(P(k ) - P(k))) 

= E(e 0i e 0k ) + z'oE{P(i) - p(j))(p(k) ~ P(k)) z o 
- Z oE((P(i) - P(i))eok) ~ E( s 0i(P[k) ~ P(k)) ) z 0 
= <T ik (l + z b(Z l Z) _1 z 0 ) ( 7 - 36 ) 

Note that E((P (i) - P (l) ) £ ok) = 0 since p (i) = (Z'Z) _1 Z'e (0 + P { ,) is independent 

of£ 0 * Asi milar rcsuJt holds for £(eoi(P(jt) - 

Maximum likelihood estimators and their distributions can be obtained when 
the errors E have a normal distribution. 


Multivariate Multiple Regression 395 


Result 7.10. Let the multivariate multiple regression model in (7-23) hold with full 
rank (Z) - r + 1, « > (r + 1) + m, and let the errors E have a normal distribu¬ 
tion. Then 

p = (Z'ZJ^Z'Y 

is the maximum likelihood estimator of p and ft has a normal distribution with 
E(f$) = P and Cov(p^, P(k)) = 07 jt(Z'Z) 1 . Also, p is independent of the max¬ 
imum likelihood estimator of the positive definite X given by 

X-±c'e = ±(Y-Zj8)'(Y-zj8) 

and 

nX is distributed as W^„- r _i(X) 

The maximized likelihood L (/i, X) = (27r)“ m "/ 2 |X|~' I / 2 e“ m/1 / 2 . 

Proof. (See website: www.prenhall.com/statistics) ■ 

Result 7.10 provides additional support for using least squares estimates. 

When the errors are normally distributed, P and n~ l E'E are the maximum likeli¬ 
hood estimators of P and X, respectively. Therefore, for large samples, they have 
nearly the smallest possible variances. 

Comment. The multivariate multiple regression model poses no new computa¬ 
tional problems. Least squares (maximum likelihood) estimates, p^ = (Z'Z) Z'yp-), 
are computed individually for each response variable. Note, however, that the model 
requires that the same predictor variables be used for all responses. 

Once a multivariate multiple regression model has been fit to the data, it should 
be subjected to the diagnostic checks described in Section 7.6 for the single-response 
model. The residual vectors [e ;1 , • ■ •, ey ;n ] can be examined for normality or 

outliers using the techniques in Section 4.6. 

The remainder of this section is devoted to brief discussions of inference for the 
normal theory multivariate multiple regression model. Extended accounts of these 
procedures appear in [2] and [18]. 
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Under // 0 : /} (2 ) = 0, Y = Zi/3 (1 ) + € and the likelihood ratio test of is base: 
on the quantities involved in the 

extra sum of squares and cross products 

= (Y - Z,/3 (1) )'(Y - zj/Sd,) - (Y - Zp)'(Y - zp) 

= «(2j - X) 

where fi (l) = (ZiZ^ZjY and Xj = n -1 (Y - Z^ (1) )'(Y - Z,/3 (1) ). 

From Result 7.10, the likelihood ratio, A, can be expressed in terms of generalize 
variances: 

maxL ( ^ (I) ,2) L( « ^ /,$,w 


ma xL(B,X) 




Equivalently, Wilks’ lambda statistic 




can be used. 


Result 7.1 I. Let the multivariate multiple regression model of (7-23) hold with 2 
of full rank r + 1 and (r + 1) + m < n. Let the errors £ be normally distributed! 
Under H 0 : fi( 2 ) = 0, nX is distributed as W^ in _ r _i(X) independently of n(Xi - X| 
which, in turn, is distributed as W p>r - g (X). The likelihood ratio test of Hq is equivalen| 
to rejecting H$ for large values of f 


-2 In A = —n In 


= -nln- 


|ni + n(£i 


For n large, 5 the modified statistic 


has, to a close approximation, a chi-square distribution with m(r - q) d.f. 

Proof. (See Supplement 7 A.) ■ 

If Z is not of full rank, but has rank + 1, then /3 = (Z'Z) Z'Y, where;- J 
(Z'Z)~ is the generalized inverse discussed in [22]. (See also Exercise 7.6.) The 
distributional conclusions stated in Result 7.11 remain the same, provided thatri| 
replaced by and q + 1 by rank (Zi). However, not all hypotheses concerning p 
can be tested due to the lack of uniqueness in the identification of caused by th 
linear dependencies among the columns of Z. Nevertheless, the generalized inve 
allows all of the important MANOVA models to be analyzed as special cases of 
multivariate multiple regression model. 

5 Technically, both n — r and n - m should also be large to obtain a good chi-square approximate 
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Example 7.9 (Testing the importance of additional predictors with a multivariate 
response) The service in three locations of a large restaurant chain was rated 
according to two measures of quality by male and female patrons. The first service- 
quality index was introduced in Example 7.5. Suppose we consider a regression model 
that allows for the effects of location, gender, and the location-gender interaction on 
both service-quality indices. The design matrix (see Example 7.5) remains the same 
for the two-response situation. We shall illustrate the test of no location-gender inter¬ 
action in either response using Result 7.11. A computer program provides 


residual sum of squares \ _ 
and cross products ) 


2977.39 

1021.72 


extra sum of squares \ _ 


and cross products 


= n(X, - X) = 


441.76 246.16 
246.16 366.12 


Let be the matrix of interaction parameters for the two responses. Although 
the sample size n = 18 is not large, we shall illustrate the calculations involved in 
the test of H 0 : /3 (2 ) = 0 given in Result 7.11. Setting a = .05, we test H 0 by referring 


n /"j 1 ~ r i + + 1) I® 


| nX + n(Xi - X) 


= - 18 - 5 - 1 — j(2 -5 + 3 + 1) In(.7605) = 3.28 

to a chi-square percentage point with m(r x ~ q x ) = 2(2) = 4 d.f Since 3.28 < ^(-^5) = 
9.49, we do not reject // 0 at the 5% level. The interaction terms are not needed. ■ 

Information criterion are also available to aid in the selection of a simple but 
adequate multivariate multiple regresson model. For a model that includes d 
predictor variables counting the intercept, let 


Xd — — (residual sum of squares and cross products matrix) 

Then, the multivariate multiple regression version of the Akaike’s information 
criterion is 

AIC = n ln(| Xd |) - 2p X d 

This criterion attempts to balance the generalized variance with the number of 
parameters. Models with smaller AIC’s are preferable. 

In the context of Example 7.9, under the null hypothesis of no interaction terms, 
we have n = 18, p =2 response variables, and d = 4 terms, so 


AIC = n ln(| X |) - 2p X d = 18 In 


m 


3419.15 1267.88 
1267.88 2417.07 


-2x2X4 


= 18 X In(20545.7) - 16 = 162.75 


More generally, we could consider a null hypothesis of the form H 0 : cp = r 0 . 
where C is (r - q) X (r + 1) and is of full rank (r - q). For the choices 
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C = [O : I 1 and T 0 = 0, this null hypothesis becomes Ho'. Cfi = P( 2 ) = 0, 

the case considered earlier. It can be shown that the extra sum of squares and cross 
products generated by the hypothesis H 0 is 

ji(X, - X) = (c/8 - r 0 )'(c(z'z)' , c')' 1 (c/8 - r 0 ) 

Under the miH hypothesis, the statistic n(Xi - X) is distributed as W r - q (X) inde¬ 
pendently of X. This distribution theory can be employed to develop a test of 
H 0 : c/} = r 0 similar to the test discussed in Result 7.11. (See, for example, [18].) 


Other Multivariate Test Statistics 

Tests other than the likelihood ratio test have been proposed for testing H 0 : /J (2 ) = 0 
in the multivariate multiple regression model. 

Popular computer-package programs routinely calculate four multivariate test 
statistics. To connect with their output, we introduce some alternative notation. Let 
E be the p X p error , or residual, sum of squares and cross products matrix 

E = rii 

that results from fitting the full model. The p X p hypothesis , or extra, sum of 
squares and cross-products matrix 

H = n(t } - X) 


The statistics can be defined in terms of E and H directly, or in terms of 
the nonzero eigenvalues v\ ^ Vi — ■ • — Vs of HE 1 , where s = min(/?,r — q)> 
Equivalently, they are the roots of [ (Xj ~ X) ~ vX I = O-The definitions are 

1 |E | 

W,lks’lambda = n TTr) ;= |E - +H -| 

Pillai’s trace = — = tr [H(H + E) _l ] 

" 1 + Vi 

S 

Hotelling-Lawley trace = X ^ = tr[HE _1 ] 

i=) 


Roy’s greatest root = 


Roy’s test selects the coefficient vector a so that the univariate E-statistic based on a 
a' Yy has its maximum possible value. When several of the eigenvalues 17 , are moder¬ 
ately large, Roy’s test will perform poorly relative to the other three. Simulation 
studies suggest that its power will be best when there is only one large eigenvalue. 

Charts and tables of critical values are available for Roy’s test. (See [21] and 
[17].) Wilks’ lambda, Roy’s greatest root, and the Hotelling-Lawley trace test are 
nearly equivalent for large sample sizes. 

If there is a large discrepancy in the reported E-values for the four tests, the 
eigenvalues and vectors may lead to an interpretation. In this text, we report Wilks’ 
lambda, which is the likelihood ratio test. 
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Predictions from Multivariate Multiple Regressions 

Suppose the model Y = z/3 + £, with normal errors £, has been fit and checked for 
any inadequacies. If the model is adequate, it can be employed for predictive purposes. 

One problem is to predict the mean responses corresponding to fixed values zo 
of the predictor variables. Inferences about the mean responses can be made using 
the distribution theory in Result 7.10. From this result, we determine that 

pz 0 is distributed as Af m (/}'z 0 ,Zo(Z'Z) _ 1 z 0 X) 
and 

nX is independently distributed as W„_ r _ 1 ( > X) 

The unknown value of the regression function at z 0 is /Jz 0 . So, from the discussion 
of the r 2 -statistic in Section 5.2, we can write 


Pjo - ft' z o V 

Vz^Z'Z/^Zo/ 


n — r — 1 


/3'zp - /?'z 0 

Vz^z'zr^o 


and the 100(1 — «)% confidence ellipsoid for /8 z 0 is provided by the inequality 


(pzo - p zo y _ t X ) (p zo - P Z 0 ) 

- \f m ( n -r-1) 


=£ ztfZ'ZPzo 


n — r — m 


F m ,„_ r _ m («) (7-40) 


where F m>n - r - m (a) is the upper (100o;)th percentile of an E-distribution with m and 
n — r — m d.f. 

The 100(1 — ot)% simultaneous confidence intervals for E(Yi) = z 'oP(i) are 


zo fim ± 


m(n - r — 1) 
n — r — m 


Fm,n-r-m(<x) \U o(Z'Z) 1 Z 0 


\n — r — 1 / 

i = 1,2,.(7-41) 


where p^ is the ith column of p and <j n is the ith diagonal element of X. 

The second prediction problem is concerned with forecasting new responses 
Y 0 = P Zq + e Q at z 0 . Here e Q is independent of £. Now, 

Y 0 - p z Q = (P - P)'z 0 + £q is distributed as Af m (0, (1 + Zo(Z'Z) _1 z 0 )X) 

independently of nX, so the 100(1 - a)% prediction ellipsoid for Y 0 becomes 




(Y 0 - P’z 0 ) 


s (1 + z’oiZ’Zy'zo) 


m(n — r — 1) 


n - r - m 


F m , n _ r - m (<*) (7-42) 


The 100(1 — a)% simultaneous prediction intervals for the individual responses Yqi are 
z 'oP(i) ± yj ( m n ( ”_~ r ~„ - )(«) yj( 1 + z^(Z'Z)~ 1 z 0 )(-- _ ” — 
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where a ih and F m ^ r - m (a) are the same quantities appearing in (7-41). Com¬ 
paring (7-41) and (7-43), we see that the prediction intervals for the actual values of 
the response variables are wider than the corresponding intervals for the expected 
values. The extra width reflects the presence of the random error e 0/ . 

Example 7.10 (Constructing a confidence ellipse and a prediction ellipse for bivariate 
responses) A second response variable was measured for the computer-requirement’ 
problem discussed in Example 7.6. Measurements on the response Y 2 > disk? 
input/output capacity, corresponding to the Z\ and z 2 values in that example were 

y 2 = [301.8,396.1,328.2,307.4,362.4,369.5,229.1] 

Obtain the 95% confidence ellipse for /?z 0 and the 95% prediction ellipse for 
Y'o = [Y 0 uY 02 ] for a site with the configuration z' 0 = [1,130,7.5]. 

Computer calculations provide the fitted equation 

Yi = 14.14 + 2.25Z] + 5.67z 2 

with s = 1.812. Thus, ^( 2 ) = [14.14,2.25,5.67]. From Example 7.6, 

= [8.42,1.08,42], z' 0 p (1) = 151.97, and zJ(Z'Z)' 1 z 0 = .34725 
We find that 

*00(2) = 14.14 + 2.25(130) + 5.67(7.5) = 349.17 


£ = (y(i) - Z 0 ( 1 ) )'(y (1) - Z/}(!)) (y ( i) - Z 0 (i))'(y( 2 ) - Z0 (2) ) 
L(y( 2 ) - z 0 ( 2 ))'(ya) - z0(d) (y< 2 > - z/i( 2 ))'(y( 2 ) - z 0 ( 2 >) 

= "5.80 5.30 ~ 

5.30 13.13 


*o0(D _ f 151.97 

" = L 349 - 17 - 


ZqP(I) 

n = 7, r = 2 , and m ~ 2, a 95% confidence ellipse for p z 0 = 

(740), the set 


[ z o0 ( i) - 151.97, zq0 (2 ) - 349.17] (4) 


is, from 


1.80 5.30 T 1 rzo0 ( i) ~ 151.97 

i.30 D.dJ [*00(2) ~ 349.17_ 

s (.34725) ) 7*2, 3 (-05) 


with ^ 2 . 3 ( 05) = 9.55. This ellipse is centered at (151.97,349.17). Its orientation and 
the lengths of the major and minor axes can be determined from the eigenvalues 
and eigenvectors of nX. 

Comparing (7-40) and (742), we see that the only change required for the 
calculation of the 95% prediction ellipse is to replace zo(Z'Z) _ 1 z 0 = *34725 with 
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Response 2 


Figure 7.5 95% confidence 
and prediction ellipses for 

_ , the computer data with two 

Response 1 r 

responses. 

1 + Zo(Z'Z) - 1 z 0 = 1.34725. Thus, the 95% prediction ellipse for Yq = [Y 0 i, * 02 ] ls 
also centered at (151.97,349.17), but is larger than the confidence ellipse. Both 
ellipses are sketched in Figure 7.5. 

It is the prediction ellipse that is relevant to the determination of computer 
requirements for a particular site with the given z 0 . ■ 



7.8 The Concept of Linear Regression 

The classical linear regression model is concerned with the association between a 
single dependent variable Y and a collection of predictor variables Z\ , z 2 , • *., z r * The 
regression model that we have considered treats Y as a random variable whose 
mean depends upon fixed values of the z/s.This mean is assumed to be a linear func¬ 
tion of the regression coefficients /%, Pi> • • ■, P r - 

The linear regression model also arises in a different setting. Suppose all the 
variables Y , Z \, Z 2 ,..., Z r are random and have a joint distribution, not necessarily 
normal, with mean vector pi and covariance matrix X . Partitioning /a 

(r+l)xi (r+l)x(r+l) 

and X in an obvious fashion, we write 


My 

a YY I 

°z y 

(ixi) 

(1x1) 1 

(IX') 

. and X ~ 

j 


Mz 

o- ZY \ 

2 zz 

('XI) 

(rXl) j 

(rxr) 


with 

°Z Y ~ {°‘YZ ] iO m YZ 2 -> - - ' ' a YZ r ] (744) 

Xzz can be taken to have full rank . 6 Consider the problem of predicting Y using the 
linear predictor = b 0 + b Y Zi + •■• + b r Z r = b 0 + b'Z (745) 

6 If X zz is not of full rank, one variable—for example, Z k —can be written as a linear combination of 
the other Z ?s and thus is redundant in forming the linear regression function Z' fi. That is, Z may be 
replaced by any subset of components whose nonsingular covariance matrix has the same rank as 2 ZZ . 
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For a given predictor of the form of (745), the error in the prediction of Y is 

prediction error = Y — bo - b{Z\ - • • - b r Z r = Y — b o — b'Z (746) 
Because this error is random, it is customary to select b 0 and b to minimize the 

mean square error = E(Y - b 0 - b'Z) 2 (747) 

Now the mean square error depends on the joint distribution of Y and Z only 
through the parameters p and X. It is possible to express the optimal linear pre¬ 
dictor in terms of these latter quantities. 


Result 7.12. The linear predictor /? 0 + P'Z with coefficients 
p = Xzz^z v, Po = Py ~ P'Vz 

has minimum mean square among ah linear predictors of the response Y. Its mean 
square error is 

E(Y - Po - p'Z) 2 = E(Y - p Y ~ tr ZY ^zz(Z ~ Mz)) 2 = °yy *“ "zy^zz^zy 
Also, p 0 + P’Z = p Y + <r'z Y Zzz( z ~ Mz) “ the linear Predictor having maxi¬ 
mum correlation with Y\ that is, 

Corr(K,/3 0 + P'Z) = maxCorr(F, b 0 + b'Z) 

6o,b 




Proof. Writing + b'Z = b 0 + b'Z + (py b'/i z ) (My b'/* z ), weget 

E(Y - bo- b'Z) 2 = E[Y -p Y ~ (b'Z - b>z) + (py - *> 0 - b> z )] 2 

= E(Y - p Y ) 2 + E{ b'(Z - p z )) 2 + (p Y ~ b 0 - b> z ) 2 

- 2£[b'(Z - pz)(Y - My)] 
= (t Y y + b'Xzzb + (Py ~ bo ~ b'/* z ) 2 — 2b'<r Z y 

Adding and subtracting (Tzy^zz^zy ^ we obtain 

E(Y - b 0 ~ b'Z) 2 = oyy ~ ^zy^zz^zy + (My - *>o “ b> z ) 2 

+ (b — Xzz^zy) ^ z z(b ~ Xzz^zk) 

The mean square error is minimized by taking b = Xzz°zy = P* making the last 
term zero, and then choosing bo = Py ~ &zz&zy) Pz ~ Po 1° make the third 
term zero. The minimum mean square error is thus cr YY - OzyXzzOzy* 

Next, we note that Cov(ho + b'Z, K) = Cov(b'Z, Y) = b'<r Z y so 

rb'ovv] 2 

[Corr(fco + b'Z, K)] 2 = foraH/, 0 ,b 

Employing the extended Cauchy-Schwartz inequality of (2-49) with B = S Z z. we 
obtain 

(b'o-zr) 2 - b'Jzzbo’zyXzzO’zy 
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[Corr(h 0 + b'Z,F)] 2 < 


2 _ ff zy^zz ff zy 


with equality for b = X zz <r Z y = p. The alternative expression for the maximum 
correlation follows from the equation ozy^zz^zy = °z yP ~ o'zy^zz^zz P ~ 
P^zzP ■ 

The correlation between Y and its best linear predictor is called the population 
multiple correlation coefficient 

/0- Z yX ZZ 0- Z y ^ 

t»( Z) = 

The square of the population multiple correlation coefficient, py( Z ), is called the 
population coefficient of determination. Note that, unlike other correlation coeffi¬ 
cients, the multiple correlation coefficient is a positive square root, so 0 ^ Py(z) ^ 1* 
The population coefficient of determination has an important interpretation. 
From Result 7.12, the mean square error in using /3 0 + p'Z to forecast Y is 


o’zy^zzP’zy — cr YY — 


O’ZY^ZZV'ZY 

O-yy 


°‘yy( 1 - Py(z)) C 7 " 49 ) 


Py(z) = 0, there is no predictive power in Z. At the other extreme, py( Z ) = 1 im¬ 
plies that Y can be predicted with no error. 


Example 7-11 (Determining the best linear predictor, its mean square error, and the 
multiple correlation coefficient) Given the mean vector and covariance matrix of F, 
> Z 2 , 


2 and X 
0 


O-yy 

<*z y 

°zy 

2zz 


■ io i 

i 

-1 

i j 

7 

3 

.-i| 

3 

2. 


determine (a) the best linear predictor p 0 + p x Z y + /3 2 Z 2 , (b) its mean square 
error, and (c) the multiple correlation coefficient. Also, verify that the mean square 
error equals o- y y(l - py(z))- 
First, 


P ~ Xzz°z} 


3 2 -1 


.4 -.6 

-.6 1.4 


Po ~ Py ~~ P pz — 5 — [1, -2] |_ o J = 3 
so the best linear predictor is p 0 + P'Z = 3 + Z\ — 2Z 2 . The mean square error is 


^ZY^ZZ^ZY — 10 ~ [ 1 ,- 1 ] 


.4 -.6 1 

-.6 1.4 -1 


= 10 - 3 = 7 
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and the multiple correlation coefficient is 

v’zy'ZzzO'ZY fy CAn 

Prm = V' <r ry ~ = Vw = - 548 

Note that cr y y(l - p\{z)) = 10(l - = 7 is the mean square error. m 

It is possible to show (see Exercise 7.5) that 

1 - PY(Z) = -ifv ■ (7-50) 

where p YY is the upper-left-hand corner of the inverse of the correlation matrix 
determined from X. 

The restriction to linear predictors is closely connected to the assumption of 
normality. Specifically, if we take 


Z 2 to be distributed as N r+1 (/i,X) 


then the conditional distribution of Y with z u z 2 , ..., z r fixed (see Result 4.6) is 

N(p<Y + 0‘Zy'£z!z(Z‘ ~~ /*Z )* or YY ~ cr ’zY^‘ZZ a ‘ZY) 

The mean of this conditional distribution is the linear predictor in Result 7.12. 
That is, 

E(Y\z\, z 2 ,. >z r ) = My + 0 zk£zz( z ~ Mz) (7-51) 

= Po + P' z 

and we conclude that E{Y\z\, z 2 > • ■ • > z r ) is the best linear predictor of Y when the 
population is N r + i(/*,X). The conditional expectation of Y in (7-51) is called the 
regression function. For normal populations, it is linear. 

When the population is not normal, the regression function E(Y | Z\, z 2 ,... ,z r ) 
need not be of the form p 0 + P’ z. Nevertheless, it can be shown (see [22]) that 
E(Y | ziy Z 2 ,. • ■, z r ), whatever its form, predicts Y with the smallest mean square 
error. Fortunately, this wider optimality among all estimators is possessed by the 
linear predictor when the population is normal. 

Result 7.13. Suppose the joint distribution of TandZ is N r+1 (/i,X). Let 


~y 

M = [Y 


and S = 


be the sample mean vector and sample covariance matrix, respectively, for a random 
sample of size n from this population. Then the maximum likelihood estimators of 
the coefficients in the linear predictor are 

P = Sz^szy, p 0 = Y - Sz^SzzZ ~ Y — p Z 
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Consequently, the maximum likelihood estimator of the linear regression function is 
Po + pz = F + siySi 1 z(z - Z) 

and the maximum likelihood estimator of the mean square error E[Y - p 0 - p* Z] 2 is 

<*YY\Z = “ ( S YY ~ s 'zySzZ*Zy) 

Proof. We use Result 4.11 and the invariance property of maximum likelihood esti¬ 
mators. [See (4-20).] Since, from Result 7.12, 

Po = My ~~ (ZzzVzy)’P z* 

P — ^ZZ&Z Y> Po 4" P' z = My °‘ZY^Zz{ z ~ Mz) 


mean square error — &yy*z — o"yy — uzy^zz^zy 
the conclusions follow upon substitution of the maximum likelihood estimators 


and X = 


<t y y ;j cr z Y 
&ZY | ^ZZ 


(^> 


and X = 


a yy 

v'zY 

.^zy 

^zz. 


It is customary to change the divisor from n to n - (r + 1) in the estimator of the 
mean square error, cryy. z = E(Y - p 0 - p' Z) 2 , in order to obtain the unbiased 
estimator 


) 

n - r - 1/ 


( s yy ~ SzySzz^zy) - 


2 (Yj -po ~ p’Zj) 
y~i _ 

n - r - 1 


Example 7.12 (Maximum likelihood estimate of the regression function—single 
response) For the computer data of Example 7.6, the n — 1 observations on Y 
(CPU time), Z x (orders), and Z 2 (add-delete items) give the sample mean vector 
and sample covariance matrix: 



150.44 “ 
130.24 
3.547 _ 



s Y y 

s zy 


467.913 

418.763 

35.983 

s = 

. 

s zy 1 

s zz 

— 

418.763 
_ 35.983 

377.200 

28.034 

28.034 

13.657 
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Assuming that Y, Z \, and Z 2 are jointly normal, obtain the estimated regression 
function and the estimated mean square error. 

Result 7.13 gives the maximum likelihood estimates 


P - S z z»zy 


.003128 

-.006422 


-.006422 

.086404 


8.7631 T1J 

5.983 J ~ 


Po = y - P'z = 150.44 - [1.079, .420] 


13024 

3.547 


= 150.44 - 142.019 


and the estimated regression function 

Po + p’z = 8.42 - l.O&i + .42*2 

The maximum likelihood estimate of the mean square error arising from the 
prediction of Y with this regression function is 


— ) ( s yy ~ s zySzz s zy) 


= I -j I 467.913 - [418.763,35.983] 


.003128 

-.006422 


-.006422 

.086404 


418.763 

35.983 


Prediction of Several Variables 

The extension of the previous results to the prediction of several responses Y u 
Y 2i ..., Y m is almost immediate. We present this extension for normal populations. 
Suppose 


is distributed as N m+r (jx, X) 


and X = 


Syy Syz 

(mXm) (mXr) 

SzY SZZ 

(rXm) (rXr) 


By Result 4.6, the conditional expectation of [ Y u Y 2 ,..., Y m ]\ given the fixed values 
Zi> • • •, z r of the predictor variables, is 

£[Y|z„Z2,.-,r r ] = Mv + 2 vz Xzz(z “ Mz) (7-53) 

This conditional expected value, considered as a function of zi, Z 2 , ■ ■ ■, z r , is called 
the multivariate regression of the vector Y on Z. It is composed of m univariate 
regressions. For instance, the first component of the conditional mean vector is 
My x + Sy lZ Szz( z ~ Mz) = Efrlzu z 2 , •. -> z r ), which minimizes the mean square 
error for the prediction of Yj. The m X r matrix /} - XyzSzz is called the matrix 
of regression coefficients. 
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The error of prediction vector 

Y - fXy — Xyz^Zz(Z — Mz) 
has the expected squares and cross-products matrix 

^yy z = E\Y — fx Y ~ SyzSzz(Z ~ Mz)] [Y - fly — S Y zSzz(Z - fxz)Y 

= Xyy — ^YZ^zzf^Yz)' “^yz^zz^zy + XyzSzzS Z zSzz(Syz )' (7“54) 

= Xyy ~ SyzSzzSzy 

Because fx and X are typically unknown, they must be estimated from a random 
sample in order to construct the multivariate linear predictor and determine expect¬ 
ed prediction errors. 

Result 7.14. Suppose Y and Z are jointly distributed as JV m+r (/t,X). Then the re¬ 
gression of the vector Y on Z is 

Po + P z = My ~ Xyz^zMz + SyzSzz z = My + ^yz^zz( z “ Mz) 

The expected squares and cross-products matrix for the errors is 

E(Y - po - JSZ)(Y - Po - pzy = S YY z = 2 yy - SyzSzzSzy 

Based on a random sample of size n, the maximum likelihood estimator of the 
regression function is 

Po + P z ~ Y + SyzSzz( z " Z) 

and the maximum likelihood estimator of X Y y*z is 

^ (n ~ 1 \ ^ x 


^YYZ ~ 


(S Y y ~ SyzSzzSzy) 


Proof. The regression function and the covariance matrix for the prediction errors 
follow from Result 4.6. Using the relationships 

Po ~ My ^yz^zzMzi P = ^yz^zz 
Po + P z — My + SyzSzz( z “ Mz) 

^yyz = Syy ~ SyzSzzSzy = S Y y ~ P^zzP 

we deduce the maximum likelihood statements from the invariance property [see 
(4-20)] of maximum likelihood estimators upon substitution of 


n _ S Y y j Syz 


■ ^ 

i ~ n 


Syy 

2yz 

fn - r 




Szy 

| S Z z 


Szy ! S7 


It can be shown that an unbiased estimator of Xyy-z is 
n _ r _ ^ (Syy - SyzSzzSzy) 

1 A i 


———7 2 (Yy - Po - PZj) (Yy - Po - pZ})' (7-55) 

n r Y y=i 
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Example 7.13 (Maximum likelihood estimates of the regression functions—two 
responses) We return to the computer data given in Examples 7.6 and 7.10. For 
Y x ~ CPU time, Y 2 = disk I/O capacity, - orders, and Z 2 = add-delete items, 
we have 


S Y y 

Syz 

_S Z Y 

Szz. 



'150.44 " 


y 

327.79 


w = " = 

z 

130.24 



3i47_ 


“ 467.913 

1148.556 418.763 

35.983' 

1148.556 

3072.491 1008.976 

140.558 

418.763 

1008.976 377.200 

28.034 

35.983 

140.558 28.034 

13.657 


Assuming normality, we find that the estimated regression function is 
£o + /* z = y + S Y zSzz( z z ) 


418.763 35.983 

1008.976 140.558_ 

f .003128 
X [--006422 

1.079(2! - 13024) H 
2.254(2! - 130.24) H 


-.006422 zi - 
.086404J \_z 2 ~ 

.420(22 - 3.547) 
5.665(2 2 - 3.547) 


Thus, the minimum mean square error predictor of T[ is 

150.44 + 1.079(2! - 130.24) + .420(z 2 ~ 3.547) = 8.42 + 1.08zi + .42z 2 
Similarly, the best predictor of Y 2 is 

14.14 + 2.25zj + 5.67 z 2 

The maximum likelihood estimate of the expected squared errors and cross- 
products matrix X Y y*z is given by 


~-)(Sv v 

-m 


- Syz^ZZ^ZY) 

467.913 1148.536 
1148.536 3072.491 


418.763 35.983 

1008.976 140.558 


= (1\ [ L043 1042 1 = H 

\7/ |_ 1.042 2.572J 


.003128 

-.006422 

894 .893 

893 2.205 


418.763 1008.976 
35.983 140.558 
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The first estimated regression function, 8.42 + 1.08ZJ + .42z 2 , and the associated 
mean square error, .894, are the same as those in Example 7.12 for the single-response 
case. Similarly, the second estimated regression function, 14.14 + 2.25Zi + 5.67z 2 , is 
the same as that given in Example 7.10. 

We see that the data enable us to predict the first response, Y 1? with smaller 
error than the second response, Y 2 . The positive covariance .893 indicates that over¬ 
prediction (underprediction) of CPU time tends to be accompanied by overpredic¬ 
tion (underprediction) of disk capacity. ■ 

Comment Result 7.14 states that the assumption of a joint normal distribu¬ 
tion for the whole collection Y u Y 2 , ..., Y m , Z u Z 2 ,... ,Z r leads to the prediction 
equations 

= Poi + P\lZ\ +'*’ + /3rl*r 
)>2 ~ P02 + 012*1 + ‘'‘ + Pr2*r 

9m 00m T Plm*l + ' " * + firmer 

We note the following: 

1. The same values, Zi, z 2 , ■ ■ • > z r are used to predict each Y}. 

2. The p ik are estimates of the (i,k )th entry of the regression coefficient matrix 

P = X Y z^zz f° r L ^ — 1- 

We conclude this discussion of the regression problem by introducing one further 
correlation coefficient. 


Partial Correlation Coefficient 


Consider the pair of errors 

Y\ - /x Yl - z^zz(Z - Mz) 

*2 “ Vy 2 ~ *y 2 zZzz(Z ~ Mz) 

obtained from using the best linear predictors to predict Y 1 and Y 2 . Their correla¬ 
tion, determined from the error covariance matrix X YYZ = X YY - Xyz^zz^zy* 
measures the association between Y y and Y 2 after eliminating the effects of Z^ 
Z2,..., Z r . 

We define the partial correlation coefficient between Y x and Y 2 , eliminating Z 1? 
Z 2 ,..., Z r , by 

PY{Y 2 Z = , ( 7 ‘ 56 / 

V^YjYrZ Vcr Y 2 J" 2 -Z 

where cr Yi Y k -z is the (/, A:)th entry in the matrix X YY -z = X YY - Xyz^zz^z y* 
corresponding sample partial correlation coefficient is 


r YyY 2 z ~ 


SYiYy z 

^YjYj-Z V 5y 2 JYZ 
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with s Yi Y k -z the (*,£)th element of S Y y “ S Y zSzzSzy-A ssuming that Y and Z have 
a joint multivariate normal distribution, we find that the sample partial correlation 
coefficient in (7-57) is the maximum likelihood estimator of the partial correlation 
coefficient in (7-56). 


Example 7.14 (Calculating a partial correlation) From the computer data i n 
Example 7.13, 




Therefore, 


r Y\Y r ‘Z ~ 


'typYZ vSy 2 Y 2 - z 


VL043 V^572 


Calculating the ordinary correlation coefficient, we obtain r Y = .96. Compar¬ 
ing the two correlation coefficients, we see that the association between Y } and Y 2 
has been sharply reduced after eliminating the effects of the variables Z on both 
responses. M 


7.9 Comparing the Two Formulations of the Regression Model 

In Sections 7.2 and 7.7, we presented the multiple regression models for one 
and several response variables, respectively. In these treatments, the predictor 
variables had fixed values z y at the fth trial. Alternatively, we can start—as 
in Section 7.8—with a set of variables that have a joint normal distribution. 
The process of conditioning on one subset of variables in order to predict values 
of the other set leads to a conditional expectation that is a multiple regression 
model. The two approaches to multiple regression are related. To show this 
relationship explicitly, we introduce two minor variants of the regression model 
formulation. 

Mean Corrected Form of the Regression Model 

For any response variable 7, the multiple regression model asserts that 

Yj = 00 + 01*1 / + ••' + PrZrj + £/ 

The predictor variables can be “centered” by subtracting their means. For instance, 
01*1/ = 0i(*i/ “ *i) + 0i*i and we can write 

Yj = (/3 0 + 0iZi + • • + P^r) + Pl(Zlj ~ Zl) + • • + P r (Zrj ~ Z r ) + Sj 
= P. + P\(Z\j “ Zl) + ••' + Pr(Zrj ~ Z r ) + 

•w_ 


(7-59) 
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with 0* - 0o + 0i*i + * “ + p r z r . The mean corrected design matrix corresponding 
to the reparameterization in (7-59) is 


1 *11 - *1 *l r " * r 

1 *21 - *1 *2r - Z r 


1 Znl - *1 


Znr *r 


where the last r columns are each perpendicular to the first column, since 

n 

2 1 (z,i - Zi) = 0, i=l,2,...,r 

i=i 


Further, setting Z c = [11 Z c2 ] with Z' 2 l = 0, we obtain 


z'z = 11 1 ZcZ n 0 

c c [z' 2 l Z' 2 Z f2 J |_0 Z' 2 Z c; 


A =(Z'Z f )''z'y 


0 (Z' 2 Zc2)' 


iy 

-i Zc 2 y 


{’Z'ci’Z'ci) J Zc2y 


That is, the regression coefficients [p ly p 2 ,are unbiasedly estimated by 
(Zc2Z c2 ) J Z' 2 y and p. is estimated by y. Because the definitions 0i, 0 2 , • • *, 0r re “ 
main unchanged by the reparameterization in (7-59), their best estimates computed 
from the design matrix Z c are exactly the same as the best estimates com¬ 
puted from the design matrix Z. Thus, setting 0' = [0j, 0 2 , -.., 0 r ], the linear 
predictor of 7 can be written as 

y = p. + P' c (z ~ z) = y + y'Z c2 (Z' c2 Z c2 y l (z - z) (7-61) 

with (z - z) = [zj ~ Zl, z 2 - z 2 . Zr ~ Zr]'. Finally, 


Var (p.) Co\(p,, p c ) 

Cov(p c ,p.) Cov(p c ) 


U 

= (Z'Z e )~V = ~n 

.» (zj 2 z c2 )“V 
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Comment The multivariate multiple regression model yields the same mean 
corrected design matrix for each response. The least squares estimates of the coeffi¬ 
cient vectors for the ith response are given by 


i = 1,2,... 


P(l) (Z' c2 Z c2 )' 1 Z' 2 y (i 


Sometimes, for even fur ther numerical stability, “standardized” input variables 
. - 5) / / j~( Tji - z,f = foi - *)M" - I)** are used. In this case, the 


( Zji -w/yjl, (*/i - = foi - *VV(« - are useu. ... m.s case, me 

slope coefficients ft in the regression model are replaced by ft = ff. V(n ~ 

The least squares estimates of the beta coefficients ft become i - 1) *^, 

i = 1,2,.., r. These relationships hold for each response in the multivariate multiple 

regression situation as well. 

Relating the Formulations 

When the variables 7, Z,, Z 2 ,..., Z, are jointly normal, the estimated predictor of 7 
(see Result 7.13) is 

ft, + p i = y + szkSzz(* “ *) = Ak + o-zv2zz(z - Az) (7-64) 

where the estimation procedure leads naturally to the introduction of centered z/s. 

Recall from the mean corrected form of the regression model that the best lin¬ 
ear predictor of 7 [see (7-61)] is 

y = ft + ft(z - Z) 

with ft = y and ft = y'Z c2 (Z' c2 Z a )~ l ■ Comparing (7-61) and (7-64), we see that 
ft = y = ft and ft = fr sin^ 7 

s zySzz = y'ZjZ'aZaf 1 ( 7 ^ 

Therefore, both the normal theory conditional mean and the classical regression 
model approaches yield exactly the same linear predictors. 

A similar argument indicates that the best linear predictors of the responses in 
the two multivariate multiple regression setups are also exactly the same. 

Example 7.15 (Two approaches yield the same linear predictor) The computer data with 

the single response 7, = CPU time were analyzed m Example 7.6 usmg the classical hn- 

ear regression model. The same data were analyzed agam in Example 7.12, assuming 
that the variables 7„ Z„ and Z 2 were jointly normal so that the best predictor of 7, is 
the conditional mean of 7, given z, and z 2 . Both approaches yielded the same predictor, 

y = 8.42 + 1.08zi + -42 z 2 “ 

’The identify in (7-65) is established by writing y = (y - yl) + X 1 s0 that 

y’Z c2 = (y - yt)'Zft + y!“ (y - + °' = (y _ ?1) ' Zc2 

Consequently, 

yZ c2 (Z' c2 Z c2 f = (y - yiyzdz' c2 z c2 )-' = (« - - ^mT 1 - 
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Although the two formulations of the linear prediction problem yield the same 
predictor equations, conceptually they are quite different. For the model in (7-3) or 
(7-23), the values of the input variables are assumed to be set by the experimenter. 
In the conditional mean model of (7-51) or (7-53), the values of the predictor vari¬ 
ables are random variables that are observed along with the values of the response 
variable(s). The assumptions underlying the second approach are more stringent, 
but they yield an optimal predictor among all choices, rather than merely among 
linear predictors. 

We close by noting that the multivariate regression calculations in either case 
can be couched in terms of the sample mean vectors y and z and the sample sums of 
squares and cross-products: 

2 (y. - y)(y. - y)' 2 (y. - y)(*/ - *)' 

Jrl . 1-1 . 

2 ( x j - *)(y. - y)' 2 (*/ - 2 )( z . - *)' 

L/-i j-i 


This is the only information necessary to compute the estimated regression coeffi¬ 
cients and their estimated covariances. Of course, an important part of regression 
analysis is model checking.This requires the residuals (errors), which must be calcu¬ 
lated using all the original data. 


7.10 Multiple Regression Models with Time Dependent Errors 

For data collected over time, observations in different time periods are often relat¬ 
ed, or autocorrelated. Consequently, in a regression context, the observations on the 
dependent variable or, equivalently, the errors, cannot be independent. As indicated 
in our discussion of dependence in Section 5.8, time dependence in the observations 
can invalidate inferences made using the usual independence assumption. Similarly, 
inferences in regression can be misleading when regression models are fit to time 
ordered data and the standard regression assumptions are used. This issue is impor¬ 
tant so, in the example that follows, we not only show how to detect the presence of 
time dependence, but also how to incorporate this dependence into the multiple re¬ 
gression model. 

Example 7.16 (Incorporating time dependent errors in a regression model) Power 
companies must have enough natural gas to heat all of their customers’ homes and 
businesses, particularly during the coldest days of the year. A major component of 
the planning process is a forecasting exercise based on a model relating the send- 
outs of natural gas to factors, like temperature, that clearly have some relationship 
to the amount of gas consumed. More gas is required on cold days. Rather than 
use the daily average temperature, it is customary to use degree heating days 


Y[Y C 

Y[Z c2 

z; 2 y c 

^c2^c2 

r -^yy j 

^YZ 

i— 
M> 

N 

2zz 
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When modeling relationships using time ordered data, regression models with 
noise structures that allow for the time dependence are often useful. Modern soft¬ 
ware packages, like SAS, allow the analyst to easily fit these expanded models. 


PANEL 7.3 SAS ANALYSIS FOR EXAMPLE 7.16 USING PROC ARIMA 


data a; 

infile T7~4.dat'; 
time = _nj 

input obsend dhd dhdlag wind xweekend; 


proc arima data = a; 
identify var = obsend crosscor = ( 
dhd dhdlag wind xweekend ); 
estimate p = (1 7) method = ml input = ( 
dhd dhdlag wind xweekend ) plot; 
estimate p = (1 7) noconstant method = ml input = ( 
dhd dhdlag wind xweekend ) plot; 

PROGRAM COMMANDS 


ARIMA Procedure 


Maximum Likelihood Estimation 


Parameter 
MU 
AR1, 1 
AR1, 2 
NUM1 
NUM2 
NUM3 
NUM4 


Estimate 

Approx. 

Std Error 

T Ratio 

Lag 

Variable 

2.12957 

13.12340 

0.16 

0 

OBSEND 

0.47008 

0.11779 

3.99 

1 

OBSEND 

0.23986 

0.11528 

2.08 

7 

OBSEND 

5.80976 

0.24047 

24.16 

0 

DHD 

1.42632 

0.24932 

5.72 

0 

DHDLAG 

1.20740 

0.44681 

2.70 

0 

WIND 

-10.10890 

6.03445 

-1.68 

0 

XWEEKEND 


Constant Estimate = 0.61770069 

Variance Estimate = 228.894028 

Std Error Estimate = 15.1292441 

AIC = 528.490321 

SBC = 543.492264 

Number of Residuals = 63 

Autocorrelation Check of Residuals 


To 

Chi 





Autocorrelations 



Lag 

Square 

DF 

Prob 







6 

6.04 

4 

0.196 

0.079 

0.012 

0.022 

0.192 

-0.127 

0.161 

12 

10.27 

10 

0.417 

0.144 

-0.067 

-0.111 

-0.056 

-0.056 

-0.108 

18 

15.92 

16 

0.45#; 

0.013 

0.106 

-0.137 

-0.170 

-0.079 

0.018 

24 

23.44 

22 

037J 

0.018 

0.004 

0.250 

-0.080 

-0.069 

-0.051 


(continues on next page) 
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PANEL 7.3 (continued) 

Autocorrelation Plot of Residuals 


Lag 

Covariance 

Correlation 

0 

228.894 

1.00000 

1 

18.194945 

0.07949 

2 

2.763255 

0.01207 

3 

5.038727 

0.02201 

4 

44.059835 

0.19249 

5 

-29.118892 

-0.12722 

6 

36.904291 

0.16123 

7 

33.008858 

0.14421 

8 

-15.424015 

-0.06738 

9 

-25.379057 

-0.11088 

10 

-12.890888 

-0.05632 

11 

-12.777280 

-0.05582 

12 

-24.825623 

-0.10846 

13 

2.970197 

0.01298 

14 

24.150168 

0.10551 

15 

-31.407314 

-0.13721 


-1 98765432101234567891 


' marks two standard errors 












Supplement 



W&S&U*'' 

*6 "f- 

-• 

. ') v\ ' }. 



The Distribution of the Likelihood 
Ratio for the Multivariate 
Multiple Regression Model 


The development in this supplement establishes Result 7.11. 

We know that nt = Y'(I - Z(Z'Z)''Z')Y and under H 0 , n% = 
Y'fl - ZiIZiZiT'Z'ilY with Y = Z,/J ( i)+£- Set P = [I ~ Z(Z'Z) Z']. 
Since 0 = [I - Z(Z'Z) -1 Z’] Z = [1 - Z(Z'Z) 'Z’][Zi i Z 2 ] = [PZj PZ 2 ] the 
columns of Z are perpendicular to P. Thus, we can write 

*i = (zp + e)T(zp + e) = e’Pe 
ni, = (Z,/5(i) + e)'WZ,ft„ + «) = e ' p i e 

where P, = I - Z^ZiZi^ZJ. We then use the Gram-Schmidt process (see Re¬ 
sult 2A.3) to construct the orthonormal vectors [gi>g2' -'8<7+i] * j 

columns of Z, . Then we continue, obtaining the orthonormal set-from [G, Z 2 J, an 
finally complete the set to n dimensions by constructing an arbitrary ort onorma 
set of n - r - 1 vectors orthogonal to the previous vectors Consequently, we have 

gl.g2.---.g?+l> .gr+2> gr-t-3’ ■•■>§<! 

from columns from columns of Z 2 arbitrary set of 
of Zj but perpendicular orthonormal 

to columns of Z, vectors orthogonal 
to columns of Z 

Let (A,e) be an eigenvalue-eigenvector pair of Z^ZJZ^ Z\. Then, since 
[Z 1 (ZiZ i r 1 Zi][Z 1 (ZiZ 1 )' 1 Zi] = Z^Z'.Z.fz,, it follows that 

Ae = Zi(ZlZi) _1 Zie = = MZrfZiZ^Zi). = A 2 e 
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and the eigenvalues of Z^ZJZ^ l Z\ are 0 or 1. Moreover, tr(Z 1 (Z , iZ 1 ) l Z\) 

= tr ((Z ^ Zj) Z^Zj) = tr ( I ^ 55 <7 4* 1 = Aj + Aj 4" * * * 4* A~+i» where 
U9+l)x<fl+l)/ _ 9 

\ x > \ 2 ^ ~ h q +] > 0 are the eigenvalues of Z^ZJZ^ 1 Z\. This shows that 

Z^ZJZj) J Z[ has q 4- 1 eigenvalues equal to 1. Now, (Z^ZIZJ^ZJ) Z x =Z l5 so 
any linear combination Zjbf of unit length is an eigenvector corresponding to the 
eigenvalue 1.The orthonormal vectors g f , € = 1,2,... ,9 + 1, are therefore eigen¬ 
vectors of Z 1 (ZiZ 1 ) l Z[ , since they are formed by taking particular linear combi¬ 
nations of the columns of Z x . By the spectral decomposition (2-16), we have 
4+ 1 

Zi(ZJZi) 1 Z\ = ^ gege- Similarly, by writing (Z(Z'Z; Z') Z = Z, we readily see 

f=i 

that the linear combination Zb^ = g e , for example, is an eigenvector of Z (Z'Z) Z' 

r+l 

with eigenvalue A = 1, so that Z (Z'Z; Z' = ^ gcgf- 

f=i 

Continuing, we have PZ = [I - Z(Z'Z) _1 Z']Z = Z- Z = 0soge = Zb?, 
f s r + l, are eigenvectors of P with eigenvalues A = 0. Also, from the way the g f , 
£ > r + 1, were constructed, Z'g f = 0, so that Pg* = g f . Consequently, these g f ’s 
are eigenvectors of P corresponding to the n — r — 1 unit eigenvalues. By the spec- 

n 

tral decomposition (2-16), P = 2 g f g' f and 

€=r + 2 

ni = cpc = 2 (e'g*)(e'g,)' = 2 VfVf 

e=r+2 £=r +2 

where, because Cov(V fi ,V jk ) = E(gie(/)fi(*)gj) = tr^gigy * 0, i * j, the E g ( = 
V* = [Vf 1? V { Vf OT ]' are independently distributed as Af m (0, X). Conse¬ 
quently, by (4-22), nX is distributed as W p i ^_ r _ 1 (2). In the same manner, 


so Pj 


Pigr 


ge t > q + 1 
0 € £ q + 1 


n 

2 gi 5 * We can write the extra sum of squares and cross products as 

€=<7+2 


n(t 1 - i) = C'(P! - P)£ = 


r+l 


2 (e'gtMe’g*)' 
€=9+2 


r+l 


2 

€=< 7+2 


where the \ € are independently distributed as N m ( 0,2). By (4-22), ^(Xj^ - X) is 
distributed as W p r _ 9 (X) independently of nX, since /2(X! - X) involves a different 
set of independent V c ’s. 

The large sample distribution for ~[n - r - 1 — |(/n - r + <? + l)]ln (| X |/| Xj |) 
follows from Result 5.2, with ^ - v 0 = m(m 4* l)/2 + wi(r + 1) - + l)/2 - 

m(q + 1) = m(r - ^) d.f The use of {n — r — 1— \ (m — r + q + l)^ instead 
of n in the statistic is due to Bartlett [4] following Box [7], and it improves the 
chi-square approximation. 
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Exercises _ 

7.1. Given the data 


Z\ 

10 

5 

7 

19 

11 

8 

y 

15 

9 

3 

25 

7 

13 


fit the linear regression model Y } = p 0 + P]Z,i + e /> /” 1 , 2 ,..., 6 . Specifically, 
calculate the least squares estimates 0, the fitted values y, the residuals e, and the 
residual sum of squares, e'e. 


7.2. Given the data 


7 19 11 18 


fit the regression model 

Y } = P iZj\ + 0 2 z ;2 + £/ , j = 1,2,...,6. 

to the standardized form (see page 412) of the variables y, Zj, and z 2 ■ From this fit, deduce 
the corresponding fitted regression equation for the original (not standardized) variables. 

7.3. (Weighted least squares estimators.) Let 

Y = Z p + e 

(flXl) (HX(r+l))(( r +J)xl) (nxl) 

where E(e ) = 0 but E(ee') = o- 2 V, with V(n X n) known and positive definite. For 
V of full rank, show that the weighted least squares estimator is 

j} w = (Z'V'Zr'Z'V'Y 

If cr 2 is unknown, it may be estimated, unbiasedly, by 

(n-r- I)' 1 X (Y - Zp w y\~\Y - Zfi w ). 

Hint: V _1 / 2 Y = (V _1/2 Z)0 + V” 1/2 e is of the classical linear regression form Y* = 
Z *p + e*, with £(e*) = 0 and E(e*£*') ~ er 2 I.Thus, = P* = (Z*Z*) *Z*'Y*. 

7 .4. Use the weighted least squares estimator in Exercise 7.3 to derive an expression for 
the estimate of the slope P in the model Yj = Pz } + e jy j = 1,2,... when 

(a) Var (e,) = <r 2 , (b) Var(e ; ) = (r 2 z h and (c) Var(e ; ) = <r 2 z). Comment on the man¬ 
ner in which the unequal variances for the errors influence the optimal choice of P w . 

7.5. Establish (7-50): Py (Z) = 1 - l/p KK - 
Hint: From (7-49) and Exercise 4.11 

, <7yy - O-zyXzZ^ZK _ l^zzl (o- y y ~ O'ZySzzO'Zx) _ jjj _ 

<ryy _ |Sa| <ryy l^zzkyy 

From Result 2A.8(c) ) <r yV = | X zz |/| X |, where <r YY is the entry of X" 1 in the first row and 
first column. Since (see Exercise223) p - V -1 ^ 2 X V and p 1 = (V '^IV — 
y 1 / 2 ( the entry in the (1,1) position of p~ l isp yy = (t yy <t Y y - 


Exercises 421 


7.6. (Generalized inverse of Z'Z) A matrix (Z'Z)~ is called a generalized inverse of Z'Z if 

Z'Z (Z'Z)~Z'Z = Z'Z. Let r\ + 1 = rank(Z) and suppose \ 2 — — A rj -n > 0 

are the nonzero eigenvalues of Z'Z with corresponding eigenvectors ej, e 2 ,..., e rj+ i. 

(a) Show that 

'i+i 

(Z'Z)- = 2 ^’e,e' 

1 = 1 

is a generalized inverse of Z'Z. 

(b) The coefficients p that minimize the sum of squared errors (y - Zp)'(y - Zp) 
satisfy the normal equations (Z'Z)p = Z'y. Show that these equations are satisfied 
for any p such that Zp is the projection of y on the columns of Z. 

(c) Show that Zp = Z (Z'Z)“Z'y is the projection of y on the columns of Z. (See Foot¬ 
note 2 in this chapter.) 

(d) Show directly that p ~ (Z'Z)~Z'y is a solution to the normal equations 
(Z'Z) [(Z'Z)~Z'y] = Z'y. 

Hint: (b) If Zp is the projection, then y — Zp is perpendicular to the columns of Z. 

(d) The eigenvalue-eigenvector requirement implies that (Z'Z) (A“ J e,) = e, for i < + 1 

and 0 = e'(Z'Z)e, for i > r } + 1. Therefore, (Z'Z) (Ar'e/JefZ'* e,e/Z'. Summing 
over i gives 


(Z'Z)(Z'Z)-Z # = Z'Z 2 AT 1 e/ef Z' 


2 e,e{ Z' = 2 z ' = IZ = z 


since e,'Z' = 0 for i > r\ -I- L 

7.7. Suppose the classical regression model is, with rank (Z) = r + 1, written as 
Y — Zj /3m + Z2 /3(2> + £ 

<" xl ) (nx( 9 +l)) (( 9 +1)xi) («x(r- 9 )) ((r-^)xl) (« x O 

where rank(Z,) = q -I- 1 and rank(Z 2 ) = r — q. If the parameters p^ 2 ) are identified 
beforehand as being of primary interest, show that a 100( 1 - a)% confidence region for 
P{2) is given by 

(/8(2)-/8 (2) )'[ZJZ 2 -Z5Z,(ZiZ ) )- , ZiZ 2 ](3 (2) - fi m ) < s 2 (r - q)F r ^ n - r - X (a) 
Hint: By Exercise 4.12, with 1 ’s and 2’s interchanged. 


C 22 = [Z' 2 Z 2 - Z^Z^ZlZjr’zSZj" 1 , where (Z'Z)- 1 


C n C 12 

c 21 c 22 


Multiply by the square-root matrix (C 22 ) l P, and conclude that (C 22 ) 1/2 (/3( 2 ) - P( 2 ))/ a ' 2 
is 7V(0,I),so that 

(P(2) ~ P(2)Y(C 22 )~\P(2) ~ P(2)) is 

7.8. Recall that the hat matrix is defined by H = Z (Z'Z) _, Z' with diagonal elements hjj. 

(a) Show that H is an idempotent matrix. [See Result 7.1 and (7-6).] 

n 

(b) Show that 0 < hjj < 1, j = 1,2and that 2 hjj = r + 1* where r is the 

/-i 

number of independent variables in the regression model. (In fact, (1 /n) < hjj < 1.) 
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(c) Verify, for the simple linear regression model with one independent variable z,that 
the leverage, h jf , is given by 

* i , 

h ii = - + ~n : 

" ZiZi-I ) 1 

;-t 

7.9. Consider the following data on one predictor variable z, and two responses y, and Y 2 : 



Determine the least squares estimates of the parameters in the bivariate straight-line re- 
gression model 

Y n = jSoi + Pn*ji + 

Yj2 = 002 + 012Z;1 + e ;2> ) ~ 1,2,3,4,5 

Also, calculate the matrices of fitted values Y and residuals € with Y - [yi ■ yi* 
Verify the sum of squares and cross-products decomposition 

Y'Y = YY + ee 

7.10. Using the results from Exercise 7.9, calculate each of the following. 

(a) A 95% confidence interval for the mean response E(Y 01 ) = 0oi + 0n*oi corre¬ 
sponding to Zoi - 0*5 

(b) A 95% prediction interval for the response y 0 i corresponding to z 0 i = 0-5 

(c) A 95% prediction region for the responses y 0 i and y 0 2 corresponding to z 0 i - 0.5 

7 11 (Generalized least squares for multivariate multiple regression.) Let A be a positive 
definite matrix, so that d)( B) = (y, - *'*,)'MYj - B'z,) is a squared statistical 
distance from the ;th observation y ) to its regression B'z, Show that the dioice 

g _ p _ (Z'Z) _1 Z'Y minimizes the sum of squared statistical distances, 2 

for any choice of positive definite A. Choices for A include S’ 1 andIL 
Hint: Repeat the steps in the proof of Result 7.10 with I" 1 replaced by A. 

7.12. Given the mean vector and covariance matrix of Y , Z\ , and Z*, 





determine each of the following. 

(a) The best linear predictor 0 O + 0i z i + 02^2 of Y 

(b) The mean square error of the best linear predictor 

(c) The population multiple correlation coefficient 

(d) The partial correlation coefficient p Y z r z 2 



1 

T 


1 
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7.13. The test scores for college students described in Example 5.5 have 

"zil [527.741 [5691.34 

z = z 2 = 54.69 , S = 600.51 126.05 

_z 3 J L 25.13J 21725 2337 23.11 _ 

Assume joint normality. 

(a) Obtain the maximum likelihood estimates of the parameters for predicting Z x from 
Z 2 and Z 3 . 

(b) Evaluate the estimated multiple correlation coefficient /?z,(z 2 ,z 3 )- 

(c) Determine the estimated partial correlation coefficient /?z,,z 2 *z 3 * 

7.14. TWenty-five portfolio managers were evaluated in terms of their performance. Suppose 
Y represents the rate of return achieved over a period of time, Z x is the manager’s atti¬ 
tude toward risk measured on a five-point scale from “very conservative” to “very risky,” 
and Z 2 is years of experience in the investment business. The observed correlation coef¬ 
ficients between pairs of variables are 


Y 

z i 

Z 2 

1.0 

-.35 

.82 

-.35 

1.0 ‘ 

-.60 

.82 

-.60 

1.0 


(a) Interpret the sample correlation coefficients r Y z x ~ ~-35 and r yz 2 ~ ~^2. 

(b) Calculate the partial correlation coefficient r Y z x -z 2 a °d interpret this quantity with 
respect to the interpretation provided for r YZl in Part a. 

The following exercises may require the use of a computer . 

7.15. Use the real-estate data in Table 7.1 and the linear regression model in Example 7.4. 

(a) Verify the results in Example 7.4. 

(b) Analyze the residuals to check the adequacy of the model. (See Section 7.6.) 

(c) Generate a 95% prediction interval for the selling price (To) corresponding to total 
dwelling size z\ = 17 and assessed value Z 2 ~ 46. 

(d) Carry out a likelihood ratio test of H 0 : 0 2 = 0 with a significance level of a = .05. 
Should the original model be modified? Discuss. 

7.16. Calculate a C p plot corresponding to the possible linear regressions involving the 

real-estate data in Table 7.1. 

7.17. Consider the Forbes data in Exercise 1.4. 

(a) Fit a linear regression model to these data using profits as the dependent variable 
and sales and assets as the independent variables. 

(b) Analyze the residuals to check the adequacy of the model. Compute the leverages 
associated with the data points. Does one (or more) of these companies stand out as 
an outlier in the set of independent variable data points? 

(c) Generate a 95% prediction interval for profits corresponding to sales of 100 (billions 
of dollars) and assets of 500 (billions of dollars). 

(d) Carry out a likelihood ratio test of H 0 :p 2 = 0 with a significance level of a = .05. 
Should the original model be modified? Discuss. 
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7 . 18 . Calculate 

(a) a C p plot corresponding to the possible regressions involving the Forbes data in 
Exercise L4. 

(b) the AIC for each possible regression. 

7 . 19 . Satellite applications motivated the development of a silver-zinc battery. Table 75 
contains failure data collected to characterize the performance of the battery during its 
life cycle. Use these data. 

(a) Find the estimated linear regression of ln(T) on an appropriate (“best”) subset of 
predictor variables. 

(b) Plot the residuals from the fitted model chosen in Part a to check the normal 
assumption. 


Table 7.5 Battery-Failure Data 

Zi 

Charge 

rate 

(amps) 

% 

Discharge 

rate 

(amps) 

-^3 

Depth of 
discharge 
(% of rated 
ampere-hours) 

Z4 

Temperature 

rc ) 

z 5 

End of 
charge 
voltage 
(volts) 

y 

Cycles to 
failure 

.375 

3.13 

60.0 

40 

2.00 

‘101 

1.000 

3.13 

76.8 

30 

1.99 

141 

1.000 

3.13 

60.0 

20 

2.00 

96 

1.000 

3.13 

60.0 

20 

1.98 

125 

1.625 

3.13 

43.2 

10 

2.01 

43 

1.625 

3.13 

60.0 

20 

2.00 

16 

1.625 

3.13 

60.0 

20 

2.02 

188 

.375 

5.00 

76.8 

10 

2.01 

10 

1.000 

5.00 

43.2 

10 

1.99 

3 

1.000 

5.00 

43.2 

30 

2.01 

386 

1.000 

5.00 

100.0 

20 

2.00 

45 

1.625 

5.00 

76.8 

10 

1.99 

2 

.375 

1.25 

76.8 

10 

2.01 

76 

1.000 

1.25 

43.2 

10 

1.99 

78 

1.000 

1.25 

76.8 

30 

2.00 

160 

1.000 

1.25 

60.0 

0 

2.00 

3 

1.625 

1.25 

43.2 

30 

1.99 

216 

1.625 

1.25 

60.0 

20 

2.00 

73 

.375 

3.13 

76.8 

30 

1.99 

314 

.375 

3.13 

60.0 

20 

2.00 

170 

Source: Selected from S. Sidik, H. Leibecki, and J. Bozek, Failure of Silver-Zinc Cells with Competing 

Failure Modes—Preliminary Data Analysis, NASA Technical Memorandum 81556 (Cleveland: Lewis Research 
Center, 1980). 


7 . 20 . Using the battery-failure data in Table 7.5, regress ln(y) on the first principal compo¬ 
nent of the predictor variables Z\, z 2 >- ■ • > Zs- (See Section 8.3.) Compare the result with 
the fitted model obtained in Exercise 7.19(a). 
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7.21. Consider the air-pollution data in Table 1.5. Let Y, = NO, and Y> - O, be the two 

responses (pollutants) corresponding to the predictor variables 2 , = wind and 
jC 2 = solar radiation. 

(a) Perform a regression analysis using only the first response K,. 

(i) Suggest and fit appropriate linear regression models. 

(ii) Analyze the residuals. 

(in) Construct a 95% prediction interval for N0 2 corresponding to z, = 10 and 
Z2 — 80 . 

(b) Perform a multivariate multiple regression analysis using both responses Y l and Y 2 . 
(0 Suggest and fit appropriate linear regression models. 

(ii) Analyze the residuals. 

(m) Construct a 95% prediction ellipse for both NO z and 0 3 for Zi = 10 and z 2 = 80. 
Compare this ellipse with the prediction interval in Part a (iii). Comment. 

7.22. Using the data on bone mineral content in Table 1 . 8 : 

(a) Perform a regression analysis by fitting the response for the dominant radius bone to 
the measurements on the last four bones. 

(i) Suggest and fit appropriate linear regression models. 

(ii) Analyze the residuals. 

(l>) bothTa^ius b^nls ariate mUltip ‘ e regression analysis by fitting the responses from 

(c) Calculate the AIC for the model you chose in (b) and for the full model. 

7.23. Using the data on the characteristics of bulls sold at auction in Table 1 . 10 : 

(a) Perform a regression analysis using the response Y, = SalePr and the predictor vari¬ 
ables Breed, YrHgt, FtFrBody, PrctFFB, Frame, BkFat, SaleHt, and SaleWt. 

( 1 ) Determine the “best” regression equation by retaining only those predictor 
variables that are individually significant. 

(ii) Using the best fitting model, construct a 95 % prediction interval for selling 

oon^n ° f P red , 1Ct ° r variable values (in the order listed above) 5,48.7, 

990,74.0,7, .18,54.2 and 1450. 

(iii) Examine the residuals from the best fitting model. 

(b> ™ at th ® a " alysis “ Par ‘ a ; us >ng the natural logarithm of the sales price as the 
response.That is, set Y, - Ln (SalePr). Which analysis do you prefer? Why? 

7.24. Using the data on the characteristics of bulls sold at auction in Table 1.10: 

(3) fj^ rrnare « res s io n analysis, using only the response VJ = SaleHt and the predic¬ 
tor variables Z, = YrHgt and Z 2 = FtFrBody. 

(i) Fit an appropriate model and analyze the residuals. 

(ii) Construct a 95% prediction interval for SaleHt corresponding to z, = 50.5 and 
Z 2 ~ 97(J. 

(b) Perform a multivariate regression analysis with the responses Y, = SaleHt and 
Y 2 = SaleWt and the predictors Z, = YrHgt and Z 2 = FtFrBody. 

(0 Fit an appropriate multivariate model and analyze the residuals. 

(ii) Construct a 95% prediction ellipse for both SaleHt and SaleWt for z, = 50.5 

and z 2 970. Compare this ellipse with the prediction interval in Part a (ii). 
Comment. x 7 



426 Chapter 7 Multivariate Linear Regression Models 
7 ' 25 ' £e 

heartbeat^bnormal blood P^^^^^y^jj^g^gre^dmiHe^to'the^hospItai 

after*an* amitOpt y Une overdose are given in Table 7.6. The two response variables 
are 

Yi = Total TC AD plasma level (TOT) 

y 2 = Amount of amitriptyline present in TCAD plasma level (AMI) 

The five predictor variables are 

2, = Gender: 1 if female. 0 if male (GEN) 

Z 2 = Amount of antidepressants taken at time of overdose (AMT) 

Z 3 = PR wave measurement (PR) 

Z 4 = Diastolic blood pressure (DIAP) 

Z 5 = QRS wave measurement (QRS) 


Table 7.6 Amitriptyline Data 


yi 

TOT 


yz 

AMI 


zi 

GEN 


Z2 

AMT 


Z3 

PR 


Z4 

DIAP 


Zs 

QRS 


3389 

3149 

1101 

653 

1131 

810 

596 

448 

896 

844 

1767 

1450 

807 

493 

1111 

941 

645 

547 

628 

392 

1360 

1283 

652 

458 

860 

722 

500 

384 

781 

501 

1070 

405 

1754 

1520 


1 

1 

0 

1 

1 

1 

1 

0 

1 

1 

1 

1 

1 

0 

0 

0 

1 


7500 

1975 

3600 

675 

750 

2500 

350 

1500 

375 

1050 

3000 

450 

1750 

2000 

4500 

1500 

3000 


220 

200 

205 

160 

185 

180 

154 

200 

137 

167 

180 

160 

135 

160 

180 

170 

180 


0 

0 

60 

60 

70 

60 

80 

70 

60 

60 

60 

64 

90 

60 

0 

90 

0 


140 

100 

111 

120 

83 

80 

98 

93 

105 

74 

80 

60 

79 

80 
100 
120 
129 


Source: See [24]. 


(a) Perform a regression analysis using only the first response 7,. 

(i) Suggest and fit appropriate linear regression models. 

(ii) Analyze the residuals. _ _ 1700 

(iii) Construct a 95% prediction interval for Total TCAD for t, U 2 
z 3 = 140, z 4 = 70, and z 5 = &5 - 

fb) Repeat Part a using the second response T>. 
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(c) Perform a multivariate multiple regression analysis using both responses Y\ and Y 2 . 

(i) Suggest and fit appropriate linear regression models. 

(ii) Analyze the residuals. 

(iii) Construct a 95% prediction ellipse for both Total TCAD and Amount of 
amitriptyline for z\ = 1, z 2 = 1200, z 3 = 140, z 4 = 70, and z$ = 85. Compare 
this ellipse with the prediction intervals in Parts a and b. Comment. 

7.26. Measurements of properties of pulp fibers and the paper made from them are contained 
in Table 7.7 (see also [19] and website: www.prenhall.com/statistics). There are n = 62 
observations of the pulp fiber characteristics, Z\ = arithmetic fiber length, z 2 “ l° n S 
fiber fraction, z 3 = fine fiber fraction, z 4 = zero span tensile, and the paper properties, 
yi = breaking length,^ = elastic modulus, y 3 = stress at failure, y 4 = burst strength. 


Table 7.7 Pulp and Paper Properites Data 


)>2 

yz 

y 4 

Zi 

Zl 

Z 3 

Z4 

BL 

EM 

SF 

BS 

AFL 

LFF 

FFF 

ZST 

21.312 

7.039 

5.326 

.932 

-.030 

35.239 

36.991 

1.057 

21.206 

6.979 

5.237 

.871 

.015 

35.713 

36.851 

1.064 

20.709 

6.779 

5.060 

.742 

.025 

39.220 

30.586 

1.053 

19.542 

6.601 

4.479 

,513 

.030 

39.756 

21.072 

1.050 

20.449 

6J95 

4.912 

.511 

-.070 

32.991 

36.570 

1.049 

16.441 

6.315 

2.997 

-.400 

-.605 

2.845 

84.554 

1.008 

16.294 

6.572 

3.017 

-.478 

-.694 

1.515 

81.988 

.998 

20.289 

7.719 

4.866 

.239 

-.559 

2.054 

8.786 

1.081 

17.163 

7.086 

3.396 

-.236 

-.415 

3.018 

5.855 

1.033 

20.289 

7.437 

4.859 

.470 

-.324 

17.639 

28.934 

1.070 

Source: See Lee [19]. 


(a) Perform a regression analysis using each of the response variables Y \, Y 2 , P 3 and Y 4 . 

(i) Suggest and fit appropriate linear regression models. 

(ii) Analyze the residuals. Check for outliers or observations with high leverage. 

(iii) Construct a 95% prediction interval for SF (P 3 ) for Z\ = .330, z 2 = 45.500, 
Zi = 20.375, z 4 = 1.010. 

(b) Perform a multivariate multiple regression analysis using all four response variables, 

Y lt Y 2 , Y 3 and Y 4> and the four independent variables, Z 1( Z 2 , Z 3 and Z 4 . 

(i) Suggest and fit an appropriate linear regression model. Specify the matrix of 
estimated coefficients fi and estimated error covariance matrix X. 

(ii) Analyze the residuals. Check for outliers. 

(iii) Construct simultaneous 95% prediction intervals for the individual responses 
Y 0i ,i = 1,2,3,4, for the same settings of the independent variables given in part 
a (iii) above. Compare the simultaneous prediction interval for Y 0 3 with the 
prediction interval in part a (iii). Comment. 

7.27. Refer to the data on fixing breakdowns in cell phone relay towers in Thble 6.20. In the 
initial design, experience level was coded as Novice or Guru. Now consider three levels 
of experience: Novice, Guru and Experienced. Some additional runs for an experienced 
engineer are given below. Also, in the original data set, reclassify Guru in run 3 as 
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Experienced and Novice in run 14 as Experienced. Keep all the other numbers for these 
two engineers the same. With these changes and the new data below, perform a multi-. 
variate multiple regression analysis with assessment and implementation times as the 
responses, and problem severity, problem complexity and experience level as the predictor 
variables. Consider regression models with the predictor variables and two factor inter-. 
action terms as inputs. (Note: The two changes in the original data set and the additional 
data below unbalances the design, so the analysis is best handled with regression 
methods.) 


Problem 

severity 

level 

Problem 

complexity 

level 

Engineer 

experience 

level 

Problem • 
assessment 
time 

Problem 

implementation 

time 

Total 

resolution 

time 

Low 

Complex 

Experienced 

5.3 

9.2 

14.5 

Low 

Complex 

Experienced 

5.0 

10.9 

15.9 

High 

Simple 

Experienced 

4.0 

8.6 

12.6 

High 

Simple 

Experienced 

4:5 

8.7 

13.2 

High 

Complex 

Experienced 

6.9 

14.9 

21.8 
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Principal Components 


8.1 Introduction 


A principal component analysis is concerned with explaining the variance^ovariance 
structure of a set of variables through a few linear combinations of these variables. Its 
general objectives are (1) data reduction and (2) interpretation. 

Although p components are required to reproduce the total system variability, 
often much of fhis variability can be accounted for by a small number k of the prin¬ 
cipal components. If so, there is (almost) as much information in the k components 
as P there Lin the original p variables. The k principal components can then replace 
the initial p variables, and the original data set. consisting of r> measurements on 
p variables, is reduced to a data set consisting of n measurements on k pnncip 

“Tnanllysis of principal components often reveals relationships that were not 
previously suspected ancfthereby allows interpretations that would not ordinarily 
result. A good example of this is provided by the stock market data discussed m 

EXa Ana e iyses of principal components are more of a means to an end rather than an 
end tn themselves, because they frequently serve as intermediate steps in much 
larger investigations. For example, principal components may be inputs to a mult p 
rewession^e^Chapter 7) or cluster analysis (see Chapter 12). Moreover (scaM 
prSTComponents are one “factoring" of the covariance matrix for the factor 

analysis model considered in Chapter 9. 


8.2 Population Principal Components 


S-* 
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with Xi>X 2 ,..., X p as the coordinate axes. The new axes represent the directions 
with maximum variability and provide a simpler and more parsimonious description 
of the covariance structure. 

As we shall see, principal components depend solely on the covariance 
matrix X (or the correlation matrix p) of X it X 2 >. -., X p . Their development does 
not require a multivariate normal assumption. On the other hand, principal 
components derived for multivariate normal populations have useful interpreta¬ 
tions in terms of the constant density ellipsoids. Further, inferences can be made 
from the sample components when the population is multivariate normal. (See 
Section 8.5.) 

Let the random vector X' = [A^, X 2y ,.., X p ] have the covariance matrix X 
with eigenvalues A] > A 2 s ■ • • > > 0. 

Consider the linear combinations 

Y\ = a'jX = ^ 11 Xi 4 012^2 + •■* + fli p X p 

Y 2 = a 2 X - \X\ 4- a 22 Xo 4 * • • + a 2p X p 

: : (8*1) 

Y p = a^X = a p} X ] 4 a p2 X 2 4 ••• 4 a pp X p 
Then, using (2-45), we obtain 

Var(y;-) - a;Xa t e = l,2,...,p (8-2) 

Cov(y /# Y k ) = aJZa* i, k = 1,2,..., p (8-3) 

The principal components are those uncorrelated linear combinations Yj, Y 2 ,..., Y p 
whose variances in (8-2) are as large as possible. 

The first principal component is the linear combination with maximum 
variance. That is, it maximizes Var(Y 3 ) = a 3 Za 3 . It is clear that Var(Y 3 ) = aJZai can 
be increased by multiplying any ai by some constant. To eliminate this indeterminacy, 
it is convenient to restrict attention to coefficient vectors of unit length. We there¬ 
fore define 

First principal component = linear combination a 3 X that maximizes 

Var(aJX) subject to alaj = 1 

Second principal component = linear combination a 2 X that maximizes 

Var(a 2 X) subject to a 2 a 2 = land 
Cov(a 3 X, a 2 X) = 0 

At the 4th step, 

ith principal component — linear combination a,' X that maximizes 
Var(a-X) subject to a/ ai = 1 and 
Cov(a|X,a*X) = 0 for kci 
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Result 8.1. Let X be the covariance matrix associated with the random vector 5 
X' = \Xi X 2 X ]. Let X have the eigenvalue-eigenvector pairs (A 1>ei ), i 

(A 2 , e 2 ),. , (A p , e p ) where Al > A 2 > • • • == A, > 0. Then the ith principal C om4 

ponent is given by Jjj 

Vj = jpJX = + ^ 2^2 + • + e ipXpi 1 ~ 1.2,..., p (8-4) ■: 

With these choices, 

Var(l/) = eJSe, = A, - 

Cov(^,n) = e-Xe* = 0 i * k (8-5)^ 

If S ome A/ are equal, the choices of the corresponding coefficient vectors, e„ and 
hence Y h are not unique. 

Proof. We know from (2-51), with B = X, that 


a'Xa 

max —;— = Ai 


(attained when a = ei) 


mt 


But eiej = 1 since the eigenvectors are normalized.Thus, 

a'Xa _ , _ \j 


a'Xa 

max—;— 
a/o a a 


A! = ^ = Var(y 1 ) 

e i e i 


Similarly, using (2-52), we get 


max = A*+i ^ 

al«i,e2,..-,e/t a a 


1,2i 


Forthechoicea = e* +1 ,withei + iei = 0,fori - 1,2.tandA: 1,2,...,p 1, 

ei+i2e* + i/ei + ie*+i = e*+iXe*+i = Var(y*+i) 

But el +1 (Xe* +1 ) = A* +1 ei +1 e t+1 = A* + i so Var (Y k+l ) = W It remains to show 
e*perpendicular toe,(thatis,.Je, -0,1* H^Cw(li.n) - H N«M» 
eigenvectors of X are orthogonal if all the eigenvalues A b A 2 ,..., A„ are distinct. If 
the eigenvalues are not all distinct, the eigenvectors corresponding to common 
eigenvalues may be chosen to be orthogonal.Therefore, for any two eigenvectors e, 
ande*,e|e* = 0,i * k. Since Xe* = A*e*,premultiplication by ej gives 

Cov(y;,y*) = e'Xe* = e'A*e* = A*eje* = 0 

for any i # k , and the proof is complete. ® 

From Result 8.1, the principal components are uncortelated and have variances 
equal to the eigenvalues of X. 

Result 8.2. Let X' = [X U X 2 ,..X p ] have covariance matrix X, with eigenvalue- 

eigenvector pairs (Ai,ej), (A 2 ,e 2 ),..., (A p ,e p ) where Aj ^ A 2 > - - p ~ 

Let Y l = eJX, K 2 = efcX,..., Y p = e^X be the principal components. Then 

o-n + Ci 2 + ••• + °PP = i Var(*,) = Ai + A 2 + - + A, = £ Var (Y,) 
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Proof. From Definition 2A.28, o- u + a 22 + • * • + a pp ~ tr(X). From (2-20) with 
A = X, we can write X = PAP' where A is the diagonal matrix of eigenvalues and 
P = [e x , e 2> ..., e p ] so that PP' = P'P = I. Using Result 2A. 12(c), we have 

tr(X) = tr(PAP') = tr(AP'P) = tr(A) = Aj + A 2 + * • • + A p 


£ Var(Ai) = tr(X) = tr(A) = £ Var(y) ■ 

1=1 f=l 

Result 8.2 says that 

Total population variance = cr u + cr 22 + + cr pp 

= Ai + A 2 + **• + K p (8-6) 

and consequently, the proportion of total variance due to (explained by) the /cth 
principal component is 


Proportion of total 
population variance 
due to/cth principal 
component 


Ai + A? + • ■ • + A n 


— k 


1,2 


If most (for instance, 80 to 90%) of the total population variance, for large p, can be 
attributed to the first one, two, or three components, then these components can 
“replace” the original p variables without much loss of information. 

Each component of the coefficient vector e' = [e n ,..., e ik ,..., e ip ] also merits 
inspection. The magnitude of e ik measures the importance of the /cth variable to the 
/th principal component, irrespective of the other variables. In particular, e ik is pro¬ 
portional to the correlation coefficient between Yj and X k . 

Result 8.3. If Yi = e[X , Y 2 = e 2 X,..., Y p = e^,X are the principal components 
obtained from the covariance matrix X, then 

PK,* = ^ /, A: = 1,2,..., p (8-8) 

are the correlation coefficients between the components Y, and the variables X k . 
Here (A 1} ei), (A 2 , e 2 ),..., (A p , e p ) are the eigenvalue-eigenvector pairs for X. 

Proof. Set a* = [0,..0,1,0,..., 0] so that X k — aJ^X and Co \(X k ,Yj) = 
Cov (aJ.X, e-X) = a^Xe/, according to (2-45). Since Xe t - = A^, Cov (X k , Yj) = a^A^ = 
A/C/jfe. Then Var(yj) = A, [see (8-5)] and Var(A^) = a kk yield 


PY„x k - 


Co\ (Y it X k ) = \je ik = e, k V\j 
''Var (Y,) VVar(^) VAj V^Tk 


i, k = 1,2,..., p 


Although the correlations of the variables with the principal components often 
help to interpret the components, they measure only the univariate contribution of 
an individual X to a component Y. That is, they do not indicate the importance of 
an A" to a component Y in the presence of the other A^s. For this reason, some 
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statisticians (see, for example, Rencher [16]) recommend that only the coefficients 
e ik t and not the correlations, be used to interpret the components. Although the co¬ 
efficients and the correlations can lead to different rankings as measures of the i m . 
portance of the variables to a given component, it is our experience that these 
rankings are often not appreciably different. In practice, variables with relatively 
large coefficients (in absolute value) tend to have relatively large correlations, so 
the two measures of importance, the first multivariate and the second univariate, 
frequently give similar results. We recommend that both the coefficients and the 
correlations be examined to help interpret the principal components. 

The following hypothetical example illustrates the contents of Results 8.1,8.2, 
and 8.3. 

Example 8.1 (Calculating the population principal components) Suppose the 
random variables A",, * 2 and X 3 have the covariance matrix 

' 1-20 
X = -2 5 0 

0 0 2 

It may be verified that the eigenvalue-eigenvector pairs are 
^ =5.83, e] = [.383,-.924,0] 

A 2 = 2.00, e 2 = [0,0,1] 

A 3 = 0.17, e£ = [.924, .383,0] 

Therefore, the principal components become 

Y x = e[X = .383A) - .924*2 

Y 2 = e 2 X = * 3 

Y 3 = e' 3 X = .924*! + 383* 2 

The variable *3 is one of the principal components, because it is uncorrelated with 
the other two variables. 

Equation (8-5) can be demonstrated from first principles. For example, 

Var(Ti) = Var(.383*! - .924* 2 ) 

= (.383) 2 Var(*j) + (-.924) 2 Var(* 2 ) 

+ 2(383)(-.924) Cov(*!,* 2 ) 

= .147(1) + .854(5) - ,708(—2) 

= 5.83 = A, 

Cov(y,,y 2 ) = Cov(.383*, - .924* 2 , * 3 ) 

= .383Cov(*!,* 3 ) - .924 Cov(* 2 , * 3 ) 

= .383(0) - .924(0) = 0 
It is also readily apparent that 

o - !! + er 2 2 + C 33 =1+5 + 2 = Ai+A 2 +A 3 = 5.83 + 2.00 + -17 
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validating Equation ( 8 - 6 ) for this example. The proportion of total variance accounted 
for by the first principal component is A 1 /(A 1 + A 2 + A 3 ) = 5.83/8 = .73. Further, the 
first two components account for a proportion (5.83 + 2)/8 = .98 of the population 
variance. In this case, the components Y 1 and Y 2 could replace the original three 
variables with little loss of information. 

Next, using ( 8 - 8 ), we obtain 

e n VTx .383 V533 „ 

‘ ‘ - 925 


PY h X 2 


euVAi 

Vo-22 


-.924V533 

V5 


-.998 


Notice here that the variable X 2 , with coefficient -.924, receives the greatest 
weight in the component Y } . It also has the largest correlation (in absolute value) 
with Yi . The correlation of X x> with Y x , .925, is almost as large as that for X 2 , indi¬ 
cating that the variables are about equally important to the first principal compo¬ 
nent. The relative sizes of the coefficients of X j and X 2 suggest, however, that X 2 
contributes more to the determination of Y\ than does X\. Since, in this case, both 
coefficients are reasonably large and they have opposite signs, we would argue that 
both variables aid in the interpretation of Ti. 

Finally, 



The remaining correlations can be neglected, since the third component is 
unimportant. ■ 


It is informative to consider principal components derived from multivariate 
normal random variables. Suppose X is distributed as 7Vp(/tt,X). We know from 
(4-7) that the density of X is constant on the /u centered ellipsoids 

(x - m)'£ _1 (* - m) = c 2 

which have axes icVXJe,, i = 1,2,...,p, where the (A,,e,) are the eigenvalue- 
eigenvector pairs of X. A point lying on the fth axis of the ellipsoid will have coordi¬ 
nates proportional to e- = [e n , e i2 ,..., e ip ] in the coordinate system that has origin 
/x and axes that are parallel to the original axes x 2 ,..., x p . It will be convenient 
to set fx - 0 in the argument that follows . 1 

From our discussion in Section 2.3 with A = X ' 1 , we can write 

c 2 = x’Z-«x = ~ (el x ) 2 + f (e’ 2 xf + ■ • • + j- (e'x) 2 
Aj A 2 A p 


’This can be done without loss of generality because the normal random vector X can always be 
translated to the normal random vector W = X - ft and = 0. However, Cov(X) = Cov(W). 
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where ej x, e 2 x,..., e^x are recognized as the principal components of x. Setting 
yi = ei X, y 2 = e 2 x, ..., y p = e^x, we have 





and this equation defines an ellipsoid (since A], A 2 ,..., \ p are positive) in a coordi¬ 
nate system with axes y { , ^ ..., y p lying in the directions of ej, e 2 ,..., e py respec¬ 
tively. If Aj is the largest eigenvalue, then the major axis lies in the direction The 
remaining minor axes lie in the directions defined by e 2 ,..., e p . 

To summarize, the principal components yf = ejx, yi = e 2 x,..., y p — e^x lie 
in the directions of the axes of a constant density ellipsoid.Therefore, any point on 
the ith ellipsoid axis has x coordinates proportional to ef - [e n , e i2> . .., e ip ] and, 
necessarily, principal component coordinates of the form [0,..., 0, y,, 0,..., 0]. 

When ^ 0, it is the mean-centered principal component y,- = e,'(x - /a) that 
has mean 0 and lies in the direction e f . 

A constant density ellipse and the principal components for a bivariate normal 
random vector with /a = 0 and p = 75 are shown in Figure 8.1. We see that the 
principal components are obtained by rotating the original coordinate axes through 
an angle 6 until they coincide with the axes of the constant density ellipse. This result 
holds for p > 2 dimensions as well. 



Figure 8.1 The constant density 
ellipse x'X M x = c 2 and the principal 
components yj, y 2 for a bivariate 
normal random vector X having 
mean 0 . 


Principal Components Obtained from Standardized Variables 

Principal components may also be obtained for the standardized variables 

Z,= 

„ (X 2 ~ M 2 ) 

Z2= ^r 

(X p - p p ) 

p ~ 


(8-9) 
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In matrix notation, 

z = (V 1/2 ) -1 (X - M ) ( 8 - 10 ) 

where the diagonal standard deviation matrix V 1/2 is defined in (2-35). Clearly, 
E(Z) = 0 and 

Cov(Z) = (V'V2(V>V = p 

by (2-37).The principal components of Z may be obtained from the eigenvectors of 
the correlation matrix p of X. All our previous results apply, with some simplifica¬ 
tions, since the variance of each Z, is unity. We shall continue to use the notation Y, 
to refer to the ith principal component and (A,-, e,) for the eigenvalue-eigenvector 
pair from either p or 2. However, the (A i( e,) derived from 2 are, in general, not the 
same as the ones derived from p. 

Result 8.4. The ith principal component of the standardized variables 
Z — [Zi , Z 2 ,..., Z p ] with Cov (Z) = p, is given by 

Y t = e;z = e;(V 1/2 )~'(X - n), i = 1 ,2,..., p 

Moreover, 

2 Var(y,) = 2 Var (Z,) = p (8-11) 

i=l /=! 

and 

Py„z k = e ik V\i i,k ~ 1,2,..., p 

In this case, (A ls e : ), (A 2 , e 2 ),..., ( X p , e^) are the eigenvalue-eigenvector pairs for 
p, with Aj > A 2 ^ • • • > x p > o. 

Proof Result 8.4 follows from Results 8.1, 8 . 2 , and 8.3, with Z 1 ,Z 2 ,...,Z„ in place 
of X u X 2 ,... t X p and p in place of X. m 

We see from (8-11) that the total (standardized variables) population variance 
is simply p, the sum of the diagonal elements of the matrix p. Using (8-7) with Z in 
place of X, we find that the proportion of total variance explained by the kth princi¬ 
pal component of Z is 

( Proportion of (standardized) \ A 

population variance due I = —, k = l, 2 ,...,p ( 8 - 12 ) 

to kih principal component ) P 

where the \ k s are the eigenvalues of p. 

Example 8.2 (Principal components obtained from covariance and correlation matrices 
are different) Consider the covariance matrix 
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and the derived correlation matrix 


The eigenvalue-eigenvector pairs from 1 are 

= 100.16, ej = [.040, .999] 

A 2 = .84, e£ = [.999, -.040] 

Similarly, the eigenvalue-eigenvector pairs from p are 

A, = 1 + p = 1.4, e[ = [.707, .707] 

A 2 =1-p= .6 , e& = [.707, -.707] 

The respective principal components become 

T, = .040*! + .999* 2 
X: y 2 = .999*! - .040* 2 


y, = .707Z, + .707Z 2 = .707 


Xi-n 


Xj M2 
10 


= .707 (*1 - Pi) + .0707(* 2 - M 2 ) 


y 2 = .707Z, - .707Z 2 = .707 


*i-Mi 


- .707 


*2 ~ M2 

10 


= .707 (*1 - Mi) - -0707 (* 2 - M 2 ) 

Because of its large variance, * 2 completely dominates the first principal component 
determined from X. Moreover, this first principal component explains a proportion 


Ai + A 2 101 

of the total population variance. 

When the variables X x and X 2 are standardized, however, the resulting 
variables contribute equally to the principal components determined from p Using 
Result 8.4, we obtain 

py„ 2l =e,,V^ = .707 VO = .837 

and _ 

py„z 2 = « 2 iVAl = .707 VL4 = .837 

In this case, the first principal component explains a proportion 

At = W = 7 

P 2 

of the total (standardized) population variance. . , - 

Most strikingly, we see that the relative importance of the variables to, tor 
instance, the first principal component is greatly affected by the standardization. 
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When the first principal component obtained from p is expressed in terms of X\ 
and X 2 , the relative magnitudes of the weights .707 and .0707 are in direct opposi¬ 
tion to those of the weights .040 and .999 attached to these variables in the principal 
component obtained from X. ■ 

The preceding example demonstrates that the principal components derived 
from X are different from those derived from p. Furthermore, one set of principal 
components is not a simple function of the other. This suggests that the standardiza¬ 
tion is not inconsequential. 

Variables should probably be standardized if they are measured on scales with 
widely differing ranges or if the units of measurement are not commensurate. For 
example, if X l represents annual sales in the $10,000 to $350,000 range and X 2 is the 
ratio (net annual income)/(total assets) that falls in the .01 to .60 range, then the 
total variation will be due almost exclusively to dollar sales. In this case, we would 
expect a single (important) principal component with a heavy weighting of X\. 
Alternatively, if both variables are standardized, their subsequent magnitudes will 
be of the same order, and X 2 (or Z 2 ) will play a larger role in the construction of the 
principal components.This behavior was observed in Example 8.2. 


Principal Components for Covariance Matrices 
with Special Structures 

There are certain patterned covariance and correlation matrices whose principal 
components can be expressed in simple forms. Suppose X is the diagonal matrix 



(8-13) 


Setting e{ = [0,..., 0,1, 0,..., 0], with 1 in the ith position, we observe that 



and we conclude that (cr ti , e f -) is the ith eigenvalue-eigenvector pair. Since the linear 
combination ej X = X x , the set of principal components is just the original set of un¬ 
correlated random variables. 

For a covariance matrix with the pattern of (8-13), nothing is gained by extracting 
the principal components. From another point of view, if X is distributed as A^(/t, X), 
the contours of constant density are ellipsoids whose axes already lie in the directions 
of maximum variation. Consequently, there is no need to rotate the coordinate system. 
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Standardization does not substantially alter the situation for the X in (8-13). In 
that case, p = I, the p X p identity matrix. Clearly, pe, = le M so the eigenvalue 1 
has multiplicity p and e- = [ 0 ,..., 0 , 1 , 0 ,..., 0 ], / = 1 , 2 ,..., p, are convenient 
choices for the eigenvectors. Consequently, the principal components determined 
from p are also the original variables Zi,...,Z p . Moreover, in this case of equal 
eigenvalues, the multivariate normal ellipsoids of constant density are spheroids. 

Another patterned covariance matrix, which often describes the correspon¬ 
dence among certain biological variables such as the sizes of living things, has the 
general form 


V 

po 2 ' 

• po 2 

par 2 

a 2 • 

• po 2 

pa 2 

po 2 ■ 



The resulting correlation matrix 


1 p • 
pi 


is also the covariance matrix of the standardized variables. The matrix in (8-15) 
implies that the variables X x ,X ly ... y X p are equally correlated. 

It is not difficult to show (see Exercise 8.5) that the p eigenvalues of the corre¬ 
lation matrix (8-15) can be divided into two groups. When p is positive, the largest is 

A] = 1 + (p - l)p (8-16) 

with associated eigenvector 

— (8-17) 

I_ |_VpVp '’V?J 

The remaining p - 1 eigenvalues are 

A2 = A3 = ’ * * = Ap = 1 ~ P 
and one choice for their eigenvectors is 

ei = „ 1 ,-^ L =, 0,...,0 

2 vrx2 vn<2 


/3~xT V2 x 3’ V2 x 3 




'(i- l)/’'"’V(i-])« > V( 


a - 1 ) 

'(<• - !)<•’ 


0 ..... 0 
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The first principal component 




is proportional to the sum of the p standarized variables. It might be regarded as an 
“index” with equal weights. This principal component explains a proportion 


Ax _ 1 + (P - l)p 
P P 



(8-18) 


of the total population variation. We see that A x /p = p for p close to 1 or p large. 
For example, if p = .80 and p = 5, the first component explains 84% of the 
total variance. When p is near 1, the last p - 1 components collectively con¬ 
tribute very little to the total variance and can often be neglected. In this special 
case, retaining only the first principal component Tj = ( 1 / Vp) [1, 1 ,. - •, 1 ]X, 
a measure of total size, still explains the same proportion (8-18) of total 
variance. 

If the standardized variables Z x , Z 2 ,..., Z p have a multivariate normal distrib¬ 
ution with a covariance matrix given by (8-15), then the ellipsoids of constant densi¬ 
ty are “cigar shaped,” with the major axis proportional to the first principal 
component Y l — ( 1 / Vp) [ 1 ,1 ,..., 1]Z. This principal component is the projection 
of Z on the equiangular line 1' = [1,1,..., l].The minor axes (and remaining prin¬ 
cipal components) occur in spherically symmetric directions perpendicular to the 
major axis (and first principal component). 


Summarizing Sample Variation by Principal Components 

We now have the framework necessary to study the problem of summarizing the 
variation in n measurements on p variables with a few judiciously chosen linear 
combinations. 

Suppose the data x l5 x 2 , ..., x n represent n independent drawings from some 
p-dimensional population with mean vector fx and covariance matrix X. These data 
yield the sample mean vector x, the sample covariance matrix S, and the sample cor¬ 
relation matrix R. 

Our objective in this section will be to construct uncorrelated linear combina¬ 
tions of the measured characteristics that account for much of the variation in the 
sample. The uncorrelated combinations with the largest variances will be called the 
sample principal components . 

Recall that the n values of any linear combination 

a{x = a n x jx + a u x }2 + ■•■ + a lp x jp , j = 1 , 2 ,...,« 

have sample mean ajx and sample variance aJSa!. Also, the pairs of values 
(ajx ; , a^Xy), for two linear combinations, have sample covariance aiSa 2 [see 
(3-36)]. 
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The sample principal components are defined as those linear combinatiotl 
which have maximum sample variance. As with the population quantities, w$,j| 
strict the coefficient vectors a, to satisfy aja, = 1. Specifically, 3| 

First sample linear combination ajxy that maximizes ^ 

principal component = the sample variance of a'jXy subject J 

toa^ai = 1 .jj 

Second sample linear combination a 2 Xy that maximizes the sample J 

principal component = variance of a^xy subject to a 2 a 2 = 1 and zero sample|j 
covariance for the pairs (ajxy, a 2 Xy) 


At the /th step, we have >|| 

l-SSfl 

/th sample linear combination a-Xy that maximizes the sample 

principal component = variance of ajxy subject to aja, = 1 and zero sample jgj 
covariance for all pairs (ajxy, a^Xy), k < i 


The first principal component maximizes aJSaj or, equivalently, 

a^Sa! 




By (2-51), the maximum is the largest eigenvalue Ai attained for the choice 
aj = eigenvector ei of S. Successive choices of ^ maximize (8-19) subject fgM B 
0 = a-Se* = aJA^e*, or a, perpendicular to e*. Thus, as in the proofs of Rcsuijl^Hj 
8.1-83, we obtain the following results concerning sample principal component^|jg|| 

IfS = is the p X p sample covariance matrix with eigenvalue-eigenvector^.;^ 
pairs (A,, e,), (A 2 , e 2 ),.... (A p , e p ), the /th sample principal component is given j- Jgg 

by . ’ | 

y i = e[x = e n x,+e i2 x 2 + - + e ip x p , i = l,2,...,p |igj 

where Aj ^ A 2 ^ ^ 0 and x is any observation on the variables ' ^ 

X\ y X 2 ,.- •, X p . Also, 


Sample variance(y*) = A*, k = 1,2,..., p 
Sample covariance (y h y k ) = 0, / * k 


In addition, 


(8-2 


Total sample variance = ^ s a = ^ + A 2 + < 
1=1 


k 1 ry 

r P„x k ~ ^r~ ’ *•k - 1 , 2 ,...,p 


and 
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We shall denote the sample principal components by y x , >* 2 ,. .., y p , irrespective 
of whether they are obtained from S or R . 2 The components constructed from S and 
R are not the same, in general, but it will be clear from the context which matrix is 
being used, and the single notation y, is convenient. It is also convenient to label the 
component coefficient vectors e, and the component variances A,- for both situations. 

The observations xy are often “centered ’ 1 by subtracting x. This has no effect on 
the sample covariance matrix S and gives the /th principal component 

y, = ej(x - x), / = 1,2. p (8-21) 

for any observation vector x. If we consider the values of the /th component 

hi = «/(*; - *). j = 1 , 2 ,..., n ( 8 - 22 ) 

generated by substituting each observation Xy for the arbitrary x in ( 8 - 21 ), then 

h = j; 2 “ *) = ~ *)) = “^ 0 = 0 ( 8 ‘ 23 ) 

That is, the sample mean of each principal component is zero. The sample variances 
are still given by the A/’s, as in (8-20). 

Example 8.3 (Summarizing sample variability with two sample principal components) 

A census provided information, by tract, on five socioeconomic variables for the 
Madison, Wisconsin, area. The data from 61 tracts are listed in Table 8.5 in the exercises 
at the end of this chapter.These data produced the following summary statistics: 


x' = [4.47, 

3.96, 

71.42, 

26.91, 

1.64] 

total 

professional 

employed 

government 

median 

population 

degree 

age over 16 

employment 

home value 

(thousands) 

(percent) 

(percent) 

(percent) 

($ 100 , 000 ) 


and 


3.397 

- 1.102 

4.306 

-2.078 

0.027 

- 1.102 

9.673 

-1.513 

10.953 

1.203 

4.306 

-1.513 

55.626 

-28.937 

-0.044 

-2.078 

10.953 

-28.937 

89.067 

0.957 

0.027 

1.203 

-0.044 

0.957 

0.319 


Can the sample variation be summarized by one or two principal components? 

2 SampIe principal components also can be obtained from X — S„, the maximum likelihood esti¬ 
mate of the covariance matrix X, if the X, are normally distributed. (See Result 4.11.) In this case, 
provided that the eigenvalues of X are distinct, the sample principal components can be viewed as 
the maximum likelihood estimates of the corresponding population counterparts. (See [l].)We shall 
not consider X because the assumption of normality is not required in this section. Also, X has eigenvalues 
[(n — l)/n]A, and corresponding eigenvectors e M where (A,-, e,) are the eigenvalue-eigenvector pairs for 
S. Thus, both S and X give the same sample principal components e'x [see (8-20)j and the same propor¬ 
tion of explained variance A,/(A x + A 2 + ■ * * + A p ). Finally, both S and X give the same sample correla¬ 
tion matrix R, so if the variables are standardized, the choice of S or X is irrelevant. 
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We find the following: 

Coefficients for the Principal Components 


Variable 


«2 (r h , Xt ) 



*5 " 

Total population 

—0.039(—.22) 

0.071(.24) 

0.188 

0.977 

-0.058 

Profession 

0.105(35) 

0.130(.26) 

-0.961 

0.171 

-0.139 

Employment (%) 

-0.492(—.68) 

0.864(.73) 

0.046 

-0.091 

0.005 

Government 

employment (%) 

0.863(.95) 

0.480(32) 

0.153 

-0.030 

0.007 

Medium home 
value 

0.009(.16) 

0.015(.17) 

-0.125 

0.082 

0.989 

Variance (A,): 

107.02 

39.67 

8.37 

2.87 

0.15 

Cumulative 
percentage of 
total variance 

67.7 

92.8 

981 

99.9 

1.000 



The first principal component explains 67.7% of the total sample variance. The 
first two principal components, collectively, explain 92.8% of the total sample van- > 
ance. Consequently, sample variation is summarized very well by two principal com-j 
ponents and a reduction in the data from 61 observations on 5 observations to 61g 
observations on 2 principal components is reasonable. j 

Given the foregoing component coefficients, the first principal component-: 
appears to be essentially a weighted difference between the percent employed byg 
government and the percent total employment. The second principal component j 
appears to be a weighted sum of the two. ■ 

As we said in our discussion of the population components, the component 
coefficients e and the correlations fp. Xk should both be examined to interpret the. 
principal components. The correlations allow for differences in the variances of 
the original variables, but only measure the importance of an individual X without 
regard to the other X*s making up the component. We notice in Example 8.3, 
however, that the correlation coefficients displayed in the table confirm the 
interpretation provided by the component coefficients. 


The Number of Principal Components 

There is always the question of how many components to retain. There is no defin- 
itive answer to this question. Things to consider include the amount of total sample: 
variance explained, the relative sizes of the eigenvalues (the variances of the sam. 
pie components), and the subject-matter interpretations of the components. In a<j 
dition, as we discuss later, a component associated with an eigenvalue near zer 
and, hence, deemed unimportant, may indicate an unsuspected linear dependenc, 
in the data. 5 
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1 Figure 8.2 A scree plot. 



A useful visual aid to determining an appropriate number of principal 
components is a scree plot. 3 With the eigenvalues ordered from largest to smallest, 
a scree plot is a plot of A f - versus i —the magnitude of an eigenvalue versus its 
number. To determine the appropriate number of components, we look for an 
elbow (bend) in the scree plot. The number of components is taken to be the 
point at which the remaining eigenvalues are relatively small and all about 
the same size. Figure 8.2 shows a scree plot for a situation with six principal 
components. 

An elbow occurs in the plot in Figure 8.2 at about i = 3. That is, the eigenvalues 
after A 2 are all relatively small and about the same size. In this case, it appears, 
without any other evidence, that two (or perhaps three) sample principal compo¬ 
nents effectively summarize the total sample variance. 

Example 8.4 (Summarizing sample variability with one sample principal component) 

In a study of size and shape relationships for painted turtles, Jolicoeur and Mosi- 
mann [11] measured carapace length, width, and height. Their data, reproduced in 
Exercise 6.18, Table 6.9, suggest an analysis in terms of logarithms. (Jolicoeur [10] 
generally suggests a logarithmic transformation in studies of size-and-shape rela¬ 
tionships.) Perform a principal component analysis. 


3 Scree is the rock debris at the bottom of a cliff. 
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The natural logarithms of the dimensions of 24 male turtles have sample mean 
vector x' = [4.725,4.478,3.703] and covariance matrix 

'11.072 8.019 8.160“ 

S = 10" 3 8.019 6.417 6.005 

8.160 6.005 6.773 _ 

A principal component analysis (see Panel 8.1 on page 447 for the output from 
the SAS statistical software package) yields the following summary: 

Coefficients for the Principal Components 

(Correlation Coefficients in Parentheses) _ 

Variable e/r?^,) *2 _ 

In (length) .683 (.99) -.159 — 

In (width) .510 (.97) -.594 .622 

In (height) .523 (.97) .788 _ -324 

Variance (i): 23.30 X KT 3 .60 X 10" 3 -36 X 1 (T 3 

Cumulative 

percentage of total 

variance 96.1 98.5 _ 


A scree plot is shown in Figure 83. The very distinct elbow in this plot occurs 
at i = 2. There is clearly one dominant principal component. 

The first principal component, which explains 96% of the total variance, has an 
interesting subject-matter interpretation. Since 

pj = .683 In (length) + .510 In (width) + .523 In (height) 

= In [ (length) * 683 (width) 510 ( height) 523 ] 
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PANEL 8.1 SAS ANALYSIS FOR EXAMPLE 8.4 USING PROC PRINCOMP. 


title 'Principal Component Analysis'; 

data turtle; 

infile 'E8-4.dat'; 

input length width height; 

xl = log(length); x2 =log(width); x3 =log{height); 

proc princomp cov data = turtle out = result; 

var xl x2 x3; 


PROGRAM COMMANDS 


24 Observations 
3 Variables 


Principal Components Analysis 


Simple Statistics 


Mean 

StD 

XI 

4.725443647 

0.105223590 

X2 

4.477573765 

0.080104466 

X3 

3.703185794 

0.082296771 


Covariance Matrix 



XI 

X2 

X3 

XI 

0.0110720040 

0.0080191419 

0.0081596480 

X2 

0.0080191419 

0.0064167255 

0.0060052707 

X3 

0.0081596480 

0.0060052707 

0.0067727585 


Total Variance = 0.024261488 


Eigenvalues of the Covariance Matrix 


Eigenvalue 

Difference 

Proportion 

Cumulative 

0.023303 

0.022705 

0.960508 

0.96051 

0.000598 

0.000238 

0.024661 

0.98517 

0.000360 


0.014832 

1.00000 


Eigenvectors 


PRIN1 

PRIN2 

PRIN3 

0.683102 

-.159479 

-.712697 

0.510220 

-.594012 

0.621953 

0.522539 . 

0.788490 

0.324401 
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the first principal component may be viewed as the In (volume)^of a box with ad- M 
justed dimensions. For instance, the adjusted height is (height ) 5 , which accounts, *g 
in some sense, for the rounded shape of the carapace. m :M 

Interpretation of the Sample Principal Components | 

The sample principal components have several interpretations. First, suppose the,Jj 
underlying distribution of X is nearly N p (n 1 2).Then the sample principal components, vi 

y- = e t -(x — x) are realizations of population principal components Yj = e-(X — / 1 ), g 

which have an A^(0, A) distribution. The diagonal matrix A has entries Aj, A 2 ,..., \ p fj 
and (A;, e^) are the eigenvalue-eigenvector pairs of X . _ .A 

Also, from the sample values x ; -, we can approximate p by x and X by S. If S is J 
positive definite, the contour consisting of all p X 1 vectors x satisfying 

(x - x)'S -1 (x - x) = c 2 (8-24) • 

estimates the constant density contour (x - *i)'X -1 (x - fi) = c 2 of the underlying ^ 
normal density. The approximate contours can be drawn on the scatter plot to indi¬ 
cate the normal distribution that generated the data. The normality assumption is 
useful for the inference procedures discussed in Section 8.5, but it is not required 
for the development of the properties of the sample principal components summa¬ 
rized in ( 8 - 20 ). 

Even when the normal assumption is suspect and the scatter plot may depart 
somewhat from an elliptical pattern, we can still extract the eigenvalues from S and ob- 
tain the sample principal components. Geometrically, the data may be plotted as n 
points in p-space. The data can then be expressed in the new coordinates, which 
coincide with the axes of the contour of (8-24). Now, (8-24) defines a hyperellipsoid 
that is centered at x and whose axes are given by the eigenvectors of S 1 or, 
equivalently, of S. (See Section 2.3 and Result 4.1, with S in place of X.) The lengths 

of these hyperellipsoid axes are proportional to /= 1 , 2 ,where 

Ai > A 2 — ' ’ * — A p ^ 0 are the eigenvalues of S. 

Because e t - has length 1, the absolute value of the ith principal component, 

| y | = | e-(x — x) |, gives the length of the projection of the vector (x - x) on the 
unit vector e,. [See (2-8) and (2-9).] Thus, the sample principal components 
y. = ej(x - x), t = 1,2,..., p, lie along the axes of the hyperellipsoid, and their 
absolute values are the lengths of the projections of x — x in the directions of the 
axes e,. Consequently, the sample principal components can be viewed as the 
result of translating the origin of the original coordinate system to x and then 
rotating the coordinate axes until they pass through the scatter in the directions of 
maximum variance. 

The geometrical interpretation of the sample principal components is illustrated 
in Figure 8.4 for p = 2. Figure 8.4(a) shows an ellipse of constant distance, centered 
at x, with Aj > A 2 . The sample principal components are well determined. They^ 
lie along the axes of the ellipse in the perpendicular directions of maximum 
sample variance. Figure 8.4(b) shows a constant distance ellipse, centered at x, wit ^ 
= A 2 . If Ai = A 2 , the axes of the ellipse (circle) of constant distance are not| 
uniquely determined and can lie in any two perpendicular directions, including t 
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(a) A| "> Aj (b) Aj — A 2 

Figure 8.4 Sample principal components and ellipses of constant distance. 


directions of the original coordinate axes. Similarly, the sample principal components 
can lie in any two perpendicular directions, including those of the original coordi¬ 
nate axes. When the contours of constant distance are nearly circular or, equiva¬ 
lently, when the eigenvalues of S are nearly equal, the sample variation is homogeneous 
in all directions. It is then not possible to represent the data well in fewer than p 
dimensions. 

If the last few eigenvalues A, are sufficiently small such that the variation in the 
corresponding £/ directions is negligible, the last few sample principal components 
can often be ignored, and the data can be adequately approximated by their repre¬ 
sentations in the space of the retained components. (See Section 8.4.) 

Finally, Supplement 8A gives a further result concerning the role of the sam¬ 
ple principal components when directly approximating the mean-centered data 


Standardizing the Sample Principal Components 

Sample principal components are, in general, not invariant with respect to changes 
in scale. (See Exercises 8.6 and 8.7.) As we mentioned in the treatment of popula¬ 
tion components, variables measured on different scales or on a common scale with 
widely differing ranges are often standardized. For the sample, standardization is 
accomplished by constructing 
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The n'X p data matrix of standardized observations 



*11 

*12 * 

• zip" 

= 

*21 

*22 * 

• z 2p 


_lni 

*/i 2 ■ 



*11 - *1 

*12 “ *2 

*i p x p 

AT 

\^22 

y^pp 

*21 - *1 

*22 ~ *2 

*2 p — X p 

A 

Vs^ 


x„i - *1 

*„2 - *2 

X np ~~ X p 

. AT 




yields the sample mean vector [see (3-24)] 


z = -(l'Z)' 


_ 1 i - i. 2 
1_ „ i-i 


h at 

A *,2 - *2 


AT =® 


_;=1 V */>P 


and sample covariance matrix [see (3-27)] 


S -TTA 


Z - — li'Z) f Z — — li'Z 


= —!—(Z - li')'(Z - 12') 

n - 1 

= _J_zz 

n - 1 


(n-l)5n (rt - 1)^12 (” jj£ip 

s l 1 V^2 '/Spp 

(n - l)s 12 (« - 1)^22 ( n ~ 

= -7 V^n V^2 *22 ^22 V^pp 

n - 1 . : *. : 

(n - 1 )j 1p (« ~ j> 2 p {" ~ 1 )*pp 

_ Vs^i Vs^p V^VJ^p s P p 


(n ~ l)si p 

VsTxV^p 

(n ~ l)s 2p 

V ^ 2 - R (S- 28 ) 


The sample principal components of the standardized observations are given by 
(8-20), with the matrix R in place of S. Since the observations are already “centered 
by construction, there is no need to write the components in the form of ( 8 - 21 ). 
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If z 1t z 2 ,..., z„ are standardized observations with covariance matrix R, the zth 
sample principal component is 

y< = = e,iZi + e i2 z 2 + •■• + e ip z p , i = 1, 2,.... p 

where (A,-, e z ) is the zth eigenvalue-eigenvector pair of R with 
> \ 2 — ^ A p ^ 0. Also, 

Sample variance (y,) = A,- z = 1,2,..., p 

Sample covariance (y z -,y*) = 0 i ^ k 

In addition, (8-29) 

Total (standardized) sample variance = tr(R) = p = Aj + A 2 + • • • + k p 
and 

r yhZk ~ i y k = 1, 2,..., p 


Using (8-29), we see that the proportion of the total sample variance explained 
by the zth sample principal component is 

(Proportion of (standardized) \ ^ 

sample variance due to zth = —- i = l,2,...,p (8-30) 

y sample principal component J ^ 

A rule of thumb suggests retaining only those components whose variances A, are 
greater than unity or, equivalently, only those components which, individually, ex¬ 
plain at least a proportion 1 /p of the total variance. This rule does not have a great 
deal of theoretical support, however, and it should not be applied blindly. As we 
have mentioned, a scree plot is also useful for selecting the appropriate number of 
components. 

Example 8.5 (Sample principal components from standardized data) The weekly 
rates of return for five stocks (JP Morgan, Citibank, Wells Fargo, Royal Dutch Shell, 
and ExxonMobil) listed on the New York Stock Exchange were determined for the 
period January 2004 through December 2005. The weekly rates of return are 
defined as (current week closing price—previous week closing price)/(previous 
week closing price), adjusted for stock splits and dividends. The data are listed in 
Table 8.4 in the Exercises. The observations in 103 successive weeks appear to be 
independently distributed, but the rates of return across stocks are correlated, 
because as one expects, stocks tend to move together in response to general 
economic conditions. 

Let x l} x 2 >..., x 5 denote observed weekly rates of return for JP Morgan, 
Citibank, Wells Fargo, Royal Dutch Shell, and ExxonMobil, respectively. Then 


x' - [.0011, .0007, .0016, .0040, .0040] 
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and 


1.000 

.632 

.511 

.115 

.155 

.632 

1.000 

.574 

.322 

.213 

.511 

.574 

1.000 

.183 

.146 

.115 

.322 

.183 

1.000 

.683 

.155 

.213 

.146 

.683 

1.000 


R - 


We note that R is the covariance matrix of the standardized observations 


Zi = 


*i ~ *\ 

V^n 


z 2 = ' 


*2 - *2 


‘ i ^5 : 


•*5 ~ *5 


The eigenvalues and corresponding normalized eigenvectors of R, determined by a 
computer, are 


X\ - 2.437, 

e; - 

[ .469, .532, 

.465, 

.387, 

.361] 

A 2 = 1.407, 

ei = 1 

[-.368,- 236,- 

-.315, 

.585, 

.606] 

A 3 - .501, 

“ 

[-.604, -.136, 

.772, 

.093, 

-.109] 

-t. 

1 ! 

ei = 

[ .363,-.629, 

.289, - 

-.381, 

.493] 

A 5 = .255, 

es = 

[ .384,-.496, 

.071, 

.595, 

-.498] 


Using the standardized variables, we obtain the first two sample principal 
components: 

y { = ejz = .469zi + .532z 2 + .465z 3 + .387z< + .361 z 5 
= e' 2 z = - .368zi - .236z 2 - .315z 3 + 5S5z 4 + -606z 5 
These components, which account for 


Ai 4- A- 


^ 100 % - 


100% = 77% 


of the total (standardized) sample variance, have interesting interpretations. The 
first component is a roughly equally weighted sum, or “index ,” of the five stocks. 

This component might be called a general stock-market component , or, simply, a 
market component . 

The second component represents a contrast between the banking stocks 
(JP Morgan, Citibank, Wells Fargo) and the oil stocks (Royal Dutch Shell, Exxon¬ 
Mobil). It might be called an industry component . Thus, we see that most of the 
variation in these stock returns is due to market activity and uncorrelated industry 
activity. This interpretation of stock price behavior also has been suggested by p| 
King [12]. r 

The remaining components are not easy to interpret and, collectively, represent ^ 
variation that is probably specific to each stock. In any event, they do not explain jj 
much of the total sample variance. 



Example 8.6 (Components from a correlation matrix with a special structure) Geneticists 
are often concerned with the inheritance of characteristics that can be measured 
several times during an animal’s lifetime. Body weight (in grams) for n = 150 
female mice were obtained immediately after the birth of their first four litters . 4 
The sample mean vector and sample correlation matrix were, respectively, 

x' = [39.88,45.08,48.11,49.95] 



1.000 

.7501 

.6329 

.6363 

.7501 

1.000 

.6925 

.7386 

.6329 

.6925 

1.000 

.6625 

.6363 

.7386 

.6625 

1.000 


The eigenvalues of this matrix are 

Ai = 3.085, A 2 = .382, A 3 = .342, and A 4 = .217 

We note that the first eigenvalue is nearly equal to 1 + (p - 1)7 = 1 + (4 — 1) (.6854) 
= 3.056, where 7 is the arithmetic average of the off-diagonal elements of R. The 
remaining eigenvalues are small and about equal, although A 4 is somewhat smaller 
than A 2 and A 3 . Thus, there is some evidence that the corresponding population 
correlation matrix p may be of the “equal-correlation” form of (8-15). This notion 
is explored further in Example 8.9. 

The first principal component 

pi = ejz = .49z, + .52z 2 + -49z 3 + .50z 4 

accounts for 100(Ai/p)% = 100(3.058/4)% = 76% of the total variance. Although 
the average postbirth weights increase over time, the variation in weights is fairly 
well explained by the first principal component with (nearly) equal coefficients. ■ 

Comment. An unusually small value for the last eigenvalue from either the sam¬ 
ple covariance or correlation matrix can indicate an unnoticed linear dependency in 
the data set. If this occurs, one (or more) of the variables is redundant and should 
be deleted. Consider a situation where x l ,x 2 , and jc 3 are subtest scores and the 
total score x 4 is the sum x x + x 2 + jc 3 . Then, although the linear combination 
e'x = [ 1 , 1 , 1 , - l]x = x x + jc 2 + x 3 - jc 4 is always zero, rounding error in the 
computation of eigenvalues may lead to a small nonzero value. If the linear 
expression relating jc 4 to (x l ,x 2 ,.x 3 ) was initially overlooked, the smallest 
eigenvalue-eigenvector pair should provide a clue to its existence. (See the discus¬ 
sion in Section 3.4, pages 131-133.) 

Thus, although “large” eigenvalues and the corresponding eigenvectors are im¬ 
portant in a principal component analysis, eigenvalues very close to zero should not 
be routinely ignored. The eigenvectors associated with these latter eigenvalues may 
point out linear dependencies in the data set that can cause interpretive and compu¬ 
tational problems in a subsequent analysis. 

4 Data courtesy of J. J. Rutledge. 
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8.4 Graphing the Principal Components 

Plots of the principal components can reveal suspect observations, as well as provide 
checks on the assumption of normality. Since the principal components are linear- 
combinations of the original variables, it is not unreasonable to expect them to be^ 
nearly normal. It is often necessary to verify that the first few principal components 
are approximately normally distributed when they are to be used as the input data 
for additional analyses. 

The last principal components can help pinpoint suspect observations. Each 
observation can be expressed as a linear combination j 

*> = (*}ei)ei + (x'e 2 )e 2 + — + (»&,)£, • - 

= >/ iCi + 5)2*2 + ■■■ + y ip e p 

of the complete set of eigenvectors , e 2 ,..., e p of S.Thus, the magnitudes of the last 
principal components determine how well the first few fit the observations. That is, 
5y i«i + $’j 2*2 + • + 5)\< 7 -iVi differs from x, by y j( fi q + ••• + y jp e p , the square of 
whose length is yj q + • • • + yj p . Suspect observations will often be such that at least 
one of the coordinates y jq ,..., y jp contributing to this squared length will be large. 
(See Supplement 8A for more general approximation results.) - 
The following statements summarize these ideas. 

1. To help check the normal assumption, construct scatter diagrams for pairs of the 
first few principal components. Also, make Q-Q plots from the sample values 
generated by each principal component. 

2. Construct scatter diagrams and Q-Q plots for the last few principal compo¬ 
nents. These help identify suspect observations. 


Example 8.7 (Plotting the principal components for the turtle data) We illustrate 
the plotting of principal components for the data on male turtles discussed in 
Example 8.4. The three sample principal components are 

5>! = .683(*i - 4.725) + .510(* 2 - 4.478) + .523(x 3 - 3.703) 

>*2 = -.159(*! - 4.725) - .594(jc 2 - 4.478) + .788(jc 3 - 3.703) 

5h = .713(Jtj - 4.725) + .622(x 2 - 4.478) + .324(* 3 - 3.703) 


where xy - In (length), x 2 = In (width), and x 3 = In (height), respectively. 

Figure 8.5 shows the Q-Q plot for 5*2 and Figure 8.6 shows the scatter plot of 
(>i, yz) f The observation for the first turtle is circled and lies in the lower right cor¬ 
ner of the scatter plot and in the upper right comer of the Q-Q plot; it may be sus¬ 
pect. This point should have been checked for recording errors, or the turtle should 
have been examined for structural anomalies. Apart from the first turtle, the scatter 
plot appears to be reasonably elliptical. The plots for the other sets of principal com¬ 
ponents do not indicate any substantial departures from normality. * 
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Figure 8.5 A Q-Q plot for the 
second principal component 5*2 from 
the data on male turtles. 


05 —Figure 8.6 Scatter plot of the 

principal components y x and 5*2 of the 
data on male turtles. 


The diagnostics involving principal components apply equally well to the 
checking of assumptions for a multivariate multiple regression model. In fact, 
having fit any model by any method of estimation, it is prudent to consider the 

Residual vector = (observation vector) - f vector of predicted^ 

\ (estimated) values ) 

or 

= 37 " P z j 7 = 1,2 ,...,n (8-31) 

(pxi) 0>X1) (pxi) 

for the multivariate linear model. Principal components, derived from the 
covariance matrix of the residuals, 

1 * 

^— p 2 (e, - £;) Ce, ~ ~ £j y (8-32) 

can be scrutinized in the same manner as those determined from a random 
sample. You should be aware that there are linear dependencies among the residuals 
from a linear regression analysis, so the last eigenvalues will be zero, within round¬ 
ing error. 
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8.5 Large Sample Inferences 

We have seen that the eigenvalues and eigenvectors of the covariance (correlation) 
matrix are the essence of a principal component analysis. The eigenvectors deter¬ 
mine the directions of maximum variability, and the eigenvalues specify the vari¬ 
ances. When the first few eigenvalues are much larger than the rest, most of the total 
variance can be “explained” in fewer than p dimensions. 

In practice, decisions regarding the quality of the principal component 
approximation must be made on the basis of the eigenvalue-eigenvector 
pairs (A,, e ( ) extracted from S or R. Because of sampling variation, these eigen¬ 
values and eigenvectors will differ from their underlying population counter¬ 
parts. The sampling distributions of A,- and e ( are difficult to derive and beyond 
the scope of this book. If you are interested, you can find some of these deriva¬ 
tions for multivariate normal populations in [1], [2],and [5]. We shall simply sum¬ 
marize the pertinent large sample results. 


Large Sample Properties of A, and e, 

Currently available results concerning large sample confidence intervals for A, and e f 
assume that the observations X lt X 2 ,..., X n are a random sample from a normal 
population. It must also be assumed that the (unknown) eigenvalues of 2 are dis¬ 
tinct and positive, so that Ai > A 2 > • * * > k p > 0. The one exception is the case 
where the number of equal eigenvalues is known. Usually the conclusions for dis¬ 
tinct eigenvalues are applied, unless there is a strong reason to believe that 2 has a 
special structure that yields equal eigenvalues. Even when the normal assumption is 
violated, the confidence intervals obtained in this manner still provide some indica¬ 
tion of the uncertainty in A, and e*. 

Anderson [2] and Girshick [5] have established the following large sample distribu¬ 
tion theory for the eigenvalues A' = [Aj,..., \ p ] and eigenvectors e lt ..., e p of S. 

1. Let A be the diagonal matrix of eigenvalues A 1? ... , A p of 2, then Vn (A - A) 
is approximately N p (0 t 2A 2 ). 


2. Let 


E, - A ; - X 


*=1 

k*i 


A k 

(\ k — A ,) 2 




then Vn (e, - e,) is approximately N p (0, E,). 

3. Each A, is distributed independently of the elements of the associated e,. 


Result 1 implies that, for n large, the A, are independently distributed. Moreover, 
A, has an approximate 7V(A„ 2A }/n) distribution. Using this normal distribution, we 
obtain E*[| A, - A,| < z{a/2)k i V2/n] = 1 - a. A large sample 100(1 - a)% confi- 
dence interval for A, is thus provided by 

_^k _ SA . S _ k _ 

(1 +zH2)V2/ii) ' (1 - z(a/2)V2fn) 


(8-33) 
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where z(a/ 2 ) is the upper 100 (a/ 2 )th percentile of a standard normal distribution. 
Bonferroni-type simultaneous 100(1 - a)% intervals for m A/s are obtained by 
replacing ziotj2) with z{a/2m). (See Section 5.4.) 

Result 2 implies that the e/s are normally distributed about the corresponding 
e/s for large samples. The elements of each e, are correlated, and the correlation 
depends to a large extent on the separation of the eigenvalues Ai, A 2 ,..■,A p (which 
is unknown) and the sample size n. Approximate standard errors for the coeffi¬ 
cients e ik are given by the square roots of the diagonal elements of (1 /n) E; where 
E, is derived from E, by substituting A/s for the A/s and e/s for the e/s. 


Example 8.8 (Constructing a confidence interval for Aj) We shall obtain a 95% con¬ 
fidence interval for A ls the variance of the first population principal component, 
using the stock price data listed in Table 8.4 in the Exercises. 

Assume that the stock rates of return represent independent drawings from 
an 7V 5 ( m , 2) population, where 2 is positive definite with distinct eigenvalues 
Ai > A 2 > • • • > A 5 > 0. Since n = 103 is large, we can use (8-33) with i — 1 to con¬ 
struct a 95% confidence interval for Ai. From Exercise 8,10, A t = .0014 and in addition, 
z(.025) = 1.96. Therefore, with 95% confidence, 


(l + 1.96 


— A, £ 


or .0011 


i' 96 v^; 


Whenever an eigenvalue is large, such as 100 or even 1000, the intervals gener¬ 
ated by (8-33) can be quite wide, for reasonable confidence levels, even though n is 
fairly large. In general, the confidence interval gets wider at the same rate that At¬ 
gets larger. Consequently, some care must be exercised in dropping or retaining 
principal components based on an examination of the A/s. 


Testing for the Equal Correlation Structure 

The special correlation structure Cov(A r i , X k ) = Verbal k p. or Corr (AT,-, X k ) = p, 
all i 9 * k , is one important structure in which the eigenvalues of 2 are not distinct 
and the previous results do not apply. 

To test for this structure, let 

1 P ■ P 

ifn-n - p 1 ' p 

Hq-P— Po “ : 

(pxp) • • 

_p p ■■■ 1_ 

and 

Hv P * Po 

A test of Ho versus H x may be based on a likelihood ratio statistic, but Lawley [14] 
has demonstrated that an equivalent test procedure can be constructed from the off- 
diagonal elements of R. 
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Lawley’s procedure requires the quantities 

h = 7^i JC* fc = 1,2,-,p; ? = p( P - 1) ?<? r,k 

i*k 

. (£ - Ifll - (1 - rf] ,8.34, 

T p - (p - 2)(1 - f) ! 

It is evident that r k is the average of the off-diagonal elements in the /cth column (or 
row) of R and r is the overall average of the off-diagonal elements. 

The large sample approximate a-level test is to reject H 0 in favor of H x if 

T = -^4 \ 22 (* -~ r ? > xUi)(P- 2W2(«) (8-35) 

(1 - r) L i<k *“1 

where x\p+D(p-2)ft ( a ) is the u PP er (100a)th percentile of a chi-square distribution 
with (p + l)(p - 2)/2 d.f. 


Example 8.9 (Testing for equicorrelation structure) From Example 8.6, the sample 
correlation matrix constructed from the n = 150 post-birth weights of female 
mice is 


1.0 

.7501 

.6329 

.6363 

.7501 

1.0 

.6925 

.7386 

.6329 

.6925 

1.0 

.6625 

.6363 

.7386 

.6625 

1.0 


We shall use this correlation matrix to illustrate the large sample test in (8-35). 
Here p = 4, and we set 


H 0 : P - Pa' 

Hi P * Po 

Using (8-34) and (8-35), we obtain 


1 P P P 
P 1 P P 
P P 1 P 
_P P P 1_ 


= i (.7501 + .6329 + .6363) = .6731, r 2 = .7271, 

? 3 = .6626, r 4 = .6791 

r = —(.7501 + .6329 + .6363 + .6925 + .7386 + .6625) = .6855 


22 ink - rf = (-7501 - .6855) 2 

i<k - 

+ (.6329 - ,6855) 2 + - + (.6625 - .6855) 

= .01277 
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2 fa ~ n = (.6731 - .6855) 2 + ••■ + (.6791 - .6855) 2 = .00245 

(4 - 1) 2 [1 - (1 - .6855) 2 ] _ 

7 4 — (4 — 2) (1 — .6855) 2 


(150 - 1) 
(1 - .6855) 2 


[.01277 - (2.1329) (.00245)] = 11.4 


Since (p + 1) (p - 2)/2 = 5(2)/2 = 5, the 5% critical value for the test in (8-35) is 
^s(.05) = 11.07. The value of our test statistic is approximately equal to the large 
sample 5% critical point, so the evidence against H 0 (equal correlations) is strong, 
but not overwhelming. 

As we saw jn Example 8.6, the smallest eigenvalues A 2 , A 3 , and A 4 are slightly 
different, with A 4 being somewhat smaller than the other two. Consequently, with 
the large sample size in this problem, small differences from the equal correlation 
structure show up as statistically significant. ■ 


Assuming a multivariate normal population, a large sample test that all vari¬ 
ables are independent (all the off-diagonal elements of X are zero) is contained in 
Exercise 8.9. 


8.6 Monitoring Quality with Principal Components 

In Section 5.6, we introduced multivariate control charts, including the quality ellipse 
and the T 2 chart. Today, with electronic and other automated methods of data collec¬ 
tion, it is not uncommon for data to be collected on 10 or 20 process variables. Major 
chemical and drug companies report measuring over 100 process variables, including 
temperature, pressure, concentration, and weight, at various positions along the pro¬ 
duction process. Even with 10 variables to monitor, there are 45 pairs for which to cre¬ 
ate quality ellipses. Clearly, another approach is required to both visually display 
important quantities and still have the sensitivity to detect special causes of variation. 

Checking a Given Set of Measurements for Stability 

Let X!, X 2 ,..., X„ be a random sample from a multivariate normal distribution with 
mean ft and covariance matrix X. We consider the first two sample principal compo¬ 
nents, y jx — ei(x y - x) and y J2 = e 2 (x ; - - x). Additional principal components 
could be considered, but two are easier to inspect visually and, of any two components, 
the first two explain the largest cumulative proportion of the total sample variance. 

If a process is stable over time, so that the measured characteristics are influ¬ 
enced only by variations in common causes, then the values of the first two principal 
components should be stable. Conversely, if the principal components remain stable 
over time, the common effects that influence the process are likely to remain con¬ 
stant. To monitor quality using principal components, we consider a two-part proce¬ 
dure. The first part of the procedure is to construct an ellipse format chart for the 
pairs of values (yj x , y j2 ) for / = 1 , 2 ,..., n. 
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By (8-20), the sample variance of the first principal component is given by the 
largest eigenvalue \ ly and the sample variance of the second principal component^ 
is the second-largest eigenvalue A 2 . The two sample components are uncorrelated 
so the quality ellipse for n large (see Section 5.6) reduces to the collection of pairs of 
possible values ( 5 >i, >* 2 ) such that 

I 1 + =£ &{*) (8-36) 

Ai A 2 


Example 8.10 (An ellipse format chart based on the first two principal components) 

Refer to the police department overtime data given in Table 5.8. Table 8.1 contains 
the five normalized eigenvectors and eigenvalues of the sample covariance matrix S. 

The first two sample components explain 82% of the total variance. 

The sample values for all five components are displayed in Table 8.2. 




Table 8.1 Eigenvectors and Eigenvalues of Sample Covariance Matrix for 
Police Department Data 


Variable 

ei 

e 2 

e 3 

e 4 


Appearances overtime (x^ 

.046 

-.048 

.629 

-.643 

.432 

Extraordinary event (jc 2 ) 

.039 

.985 

-.077 

-.151 

-.007 

Holdover hours (jc 3 ) 

-.658 

.107 

.582 

.250 

-.392 

COA hours (jc 4 ) 

.734 

.069 

.503 

.397 

-.213 

Meeting hours (jc 5 ) 

-.155 

.107 

.081 

,586 

.784 

h 

2,770,226 

1,429,206 

628,129 

221,138 

99,824 



Table 8.2 Values of the Principal Components for 
the Police Department Data 

Period 

fa 

hi 

fa 

fa 

fa 

1 


588.2 

425.8 

-189.1 


2 

-2143.7 

- 686.2 

883.6 

-565.9 

-441.5 

3 

-177.8 

-464.6 


736.3 

38.2 

4 

-2186.2 

450.5 


443.7 

-325.3 

5 

-878.6 

-545.7 

115.7 

296.4 

437.5 

6 



281.2 


142.7 

7 


66.8 


-135.5 

521.2 

8 



-1437.3 

-148.8 

61.6 

9 

132.8 

563.7 

125.3 

68.2 

611.5 


-2787.3 

-213.4 

7.8 

169.4 


11 


3936.9 


276.2 

-159.6 

12 

761.6 


-2153.6 

-418.8 

28.2 

13 

-498.3 

244.7 

966.5 

-1142.3 

182.6 

14 

2366.2 

-1193.7 

-165.5 


-344.9 

15 

1917.8 


-82.9 

-196.8 

-89.9 

16 

2187.7 

-373.8 


-84.1 




4 
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Figure 8.7 The 95% control ellipse 
based on the first two principal 
components of overtime hours. 


Let us construct a 95% ellipse format chart using the first two sample principal 
components and plot the 16 pairs of component values in Table 8.2. 

Although n = 16 is not large, we use a^(. 05) = 5.99, and the ellipse becomes 

IMLs.*, 

Ai A 2 

This ellipse centered at (0,0), is shown in Figure 8.7, along with the data. 

One point is out of control, because the second principal component for this 
point has a large value. Scanning Table 8.2, we see that this is the value 3936.9 for pe¬ 
riod 11. According to the entries of e 2 in Table 8,1, the second principal component 
is essentially extraordinary event overtime hours. The principal component approach 
has led us to the same conclusion we came to in Example 5.9. m 

In the event that special causes are likely to produce shocks to the system, the 
second part of our two-part procedure—that is, a second chart—is required. This 
chart is created from the information in the principal components not involved in 
the ellipse format chart. 

Consider the deviation vector X — /x, and assume that X is distributed as 
A/>(m>2). Even without the normal assumption, Xy — /x can be expressed as the 
sum of its projections on the eigenvectors of X 

X - /x = (X - Ax)'eie! + (X - /x)'e 2 e 2 

+ (X - M)'e 3 e 3 + • • ■ + (X - jx)'e 
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ft = + ^ 2 e 2 + ^ 3 e 3 + * * * + Yp e p 


where y. = (X - /*)'e/ is the population ith principal component centered to have 
mean O.The approximation to X - p by the first two principal components has the 

form y,e| + y 2 e 2 . This leaves an unexplained component of 

X - p - Tie, - Y 2 t 2 

Let E = [e,, e 2 ,..., e p ] be the orthogonal matrix whose columns are the eigenvec 
tors of 2. The orthogonal transformation of the unexplained part, 

Y >i hi r°i r°i 

y 2 0 Y 2 0 T 0 ' 

E-(X - M - y,e, - Y 2 e 2 ) = y 3 - 0 - 0 = y 3 = o 


so the last o-2 principal components are obtained as an orthogonal transformation 
of the approximation errors. Rather than base the T 2 chart on the approximation 
errors, we can, equivalently, base it on these last principal components. Recall that 

Var(y) = A, for i = l,2,...,p 

and Cov(V;, Y k ) = 0 for i * k. Consequently, the statistic Y( 2) 2y 2| ,Y (!) Y (2)> based 
on the last p - 2 population principal components, becomes 

II + H+ ...+ ]! (8-38) 

A3 A4 Ap 


This is just the sum of the squares of p - 2 independent standard normal variables, 
A -i/2y. and so has a chi-square distribution with p- 2 degrees of freedom. 

In terms of the sample data, the principal components and eigenvalues must be 
estimated. Because the coefficients of the linear combinations e, are also estimates, 
the principal components do not have a normal distribution even when the popula- 
tion is normal. However, it is customary to create a T -chart based on the statis ic 

r? = |l + |l + ... + ^ 

A 3 A4 kp 

which involves the estimated eigenvalues and vectors. Further, it is usual to appeal 
to the large sample approximation described by (8-38) and set the upper control 
limit of the T 2 -chart as UCL = c 2 = AfJ-2(°0- _ 

This 7 2 -statistic is based on high-dimensional data. For example, when p - M 
variables are measured, it uses the information in the 18 -dimensional space perpen¬ 
dicular to the first two eigenvectors e t and e 2 . Still, this T based on the unexpl 
variation in the original observations is reported as highly effective in picking up 
special causes of variation. 
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Example 8.11 (A T 2 -chart for the unexplained [orthogonal] overtime hours) 

Consider the quality control analysis of the police department overtime hours in 
Example 8.10, The first part of the quality monitoring procedure, the quality ellipse 
based on the first two principal components, was shown in Figure 8.7. To illustrate 
the second step of the two-step monitoring procedure, we create the chart for the 
other principal components. 

Since p - 5, this chart is based on 5 - 2 = 3 dimensions, and the upper control 
limit is ^(.OS) = 7.81. Using the eigenvalues and the values of the principal com¬ 
ponents, given in Example 8.10, we plot the time sequence of values 



where the first value is T 2 = .891 and so on. The r 2 -chart is shown in Figure 8.8. 



Figure 8.8 A r 2 -chart based on the last three principal components of overtime hours. 


Since points 12 and 13 exceed or are near the upper control limit, something has 
happened during these periods. We note that they are just beyond the period in 
which the extraordinary event overtime hours peaked. 

From Table 8.2, y 3j is large in period 12, and from Table 8.1, the large coefficients 
in e 3 belong to legal appearances, holdover, and COA hours. Was there some adjust¬ 
ing of these other categories following the period extraordinary hours peaked? ■ 


Controlling Future Values 

Previously, we considered checking whether a given series of multivariate observa¬ 
tions was stable by considering separately the first two principal components and 
then the last p - 2. Because the chi-square distribution was used to approximate 
the UCL of the T 2 -chart and the critical distance for the ellipse format chart, no fur¬ 
ther modifications are necessary for monitoring future values. 
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Example 8.12 (Control ellipse for future principal components) In Example 8.10, w| 
determined that case 11 was out of control. We drop this point and recalculate th| 
eigenvalues and eigenvectors based on the covariance of the remaining 15 observa§ 
tions,The results are shown in Table 8.3. 


Table 8.3 Eigenvectors and Eigenvalues from the 15 Stable Observations 


e 2 £ 3 £ 5.5 

Appearances overtime (x^) 
Extraordinary event (x 2 ) 
Holdover hours (x 3 ) 
COA hours (x 4 ) 
Meeting hours (x 5 ) 

.049 .629 • .304 .479 .530- 

.007 -.078 .939 -.260 -.212' 

- 662 .582 -.089 -.158 -.437-; 

.731 -503 -.123 -.336 -.291 

-.159 -081 -.058 -.752 .632 

V 

2,964,749.9 672,995.1 396,596.5 194,401.0 92,760.3 


The principal components have changed. The component consisting primarily of 
extraordinary event overtime is now the third principal component and is not includ¬ 
ed in the chart of the first two. Because our initial sample size is only 16, dropping a 
single case can make a substantial difference. Usually, at least 50 or more observa¬ 
tions are needed, from stable operation of the process, in order to set future limits. 

Figure 8.9 gives the 99% prediction (8-36) ellipse for future pain; of values for 
the new first two principal components of overtime. The 15 stable pairs of principal ^ 
components are also shown. 



Figure 8.9 A 99% ellipse 
format chart for the first two 

principal components of 
future values of overtime. 
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In some applications of multivariate control in the chemical and pharmaceutical 
industries,more than 100 variables are monitored simultaneously.These include nu¬ 
merous process variables as well as quality variables. Typically, the space orthogonal 
to the fust few principal components has a dimension greater than 100 and some of 
the eigenvalues are very small. An alternative approach (see [13]) to constructing a 
control chart, that avoids the difficulty caused by dividing a small squared principal 
component by a very small eigenvalue, has been successfully applied. To implement 
this approach, we proceed as follows. 

For each stable observation, take the sum of squares of its unexplained component 
d Uj = (*/• - X - yyiej - y /2 e 2 )'(xy - x - y^e, - y /2 e 2 ) 

Note that, by inserting EE' = I, we also have 

d Uj = (*; - x - y ,iei - y, 2 e 2 )'EE'(x y - x - y^e, - y, 2 e 2 ) = 2 m 

k =3 


which is just the sum of squares of the neglected principal components. 

Using either form, the are plotted versus j to create a control chart. The 
lower limit of the chart is 0 and the upper limit is set by approximating the distribu¬ 
tion of dhj as the distribution of a constant c times a chi-square random variable with 
v degrees of freedom. 

For the chi-square approximation, the constant c and degrees of freedom v are 
chosen to match the sample mean and variance of the dfjj , j = 1, 2,In particu¬ 
lar, we set 


and determine 


d u = ~ 2 d Vj = c V 


s d 2 = -—v 2 ( d hj - d l ) 2 = 2c ' 1 
n 1 ;=1 


& . , (dh ) 2 

c = -= and v — 2 — z — 
2 d 2 v s 2 d i 


The upper control limit is then cxl(a), where a = .05 or .01. 






Supplement 



The Geometry of the Sample 
Principal Component 
Approximation 


T ,Kic cimnlement we shall present interpretations for approximations to the data 
on the first r sample principal components. TTie interpretations of both the 
^dtonsiL scatter plot 

result that follows. We consider approximates of the form ( la,, 2 . • • •, „] 

to the mean corrected data matrix 

[xi - i, x 2 -*••••>*« “ *1' 

The error of approxim.fion b quantified as Ihe sum of the -p squared errors 

j (,, - S - -«-•,)■ 11 <*f. - * - *»>’ <8A ' ,) 

j ^ 1 

Result 8A.I Let A be any matrix with rank(A) ^ r < min(p, n). Let E f = 
[e e .... e,l, wheret is the ith eigenvector of S.TTie error of approximation sum 
of'squats in (8A-1) is minimized by the choice 

’(xi - *)'" 

a = ( * 2 7 x) E r Er = [yt.y2.WE; 

- *)’- 

so the /th column of its transpose A' is 

3/ = V/1®1 + Py2®2 " j + yjr*r 
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where 

[Pjl. Pj2, ■■■, yjr\ = [*!(*/ - *). ®2 (*; ~ *).(Xy ~ x) ]' 

are the values of the first r sample principal components for the yth unit. Moreover, 

n ^ 

2 (*; ~ * ~ »;)'(*; ~ x - ij) = (n - l)(A r+ , + + \ p ) 

1=1 

where A r +i s • > \ p are the smallest eigenvalues of S. 

Proof. Consider first any A whose transpose A' has columns a y - that are a linear 
combination of a fixed set of r perpendicular vectors u l5 u 2 ,..., u r , so that 
U = [uj, u 2 , ..., u r ] satisfies U'U = I. For fixed U, Xy - x is best approximated by 
its projection on the space spanned by u b u 2 ,..., u r (see Result 2A.3), or 

(Xj - xJ'ujU! + (Xj - x)'u 2 u 2 + •••+(*,•- x)'u r u r 

“l(Xy - X) 

= [u,,u 2 „..,u r ] u 2( x y.- x ) = UU'(X; - x) (8A-2) 

.<(*; - *)_ 

This follows because, for an arbitrary vector by, 

Xy ' X Uby = Xy - X - UU'(X; “ 5) + UU'(Xy — x) — Uby 
* (I - UU')(Xy - X) + U(U'(x, - X) - by) 
so the error sum of squares is 

(Xy - X - Uby)'(Xy — X ~ Uby) = (Xy ~ x)'(I ~ UU')(Xy ~ 5) + 0 

+ (U'(x y - x) - b y )'(U'(X; - X) - b y ) 

where the cross product vanishes because (I - UU')U = U — UU'U = 
U — U = 0. The last term is positive unless by is chosen so that by = U'(xy — x) 
and Uby = UU'(xy — x) is the projection of Xy — x on the plane. 

Further, with the choice a y = Uby = UU'(xy - x), (8A-1) becomes 

2 ( x i - * - UU'(Xy - x))'(x y - X - UU'(Xy - x)) 

1=1 

= 2 (*/ - *)'(I - UU’)(x, - x) 

7=1 

= 2 (*/ - *)'(*;' - 5) - 2 ( x i “ S)'UU'(x, - x) (8A-3) 

M i=1 

We are now in a position to minimize the error over choices of U by maximizing the 
last term in (8A-3). By the properties of trace (see Result 2A.12), 

2 ( X 1 - 5)'UU'( X ; - *) = 2 tr t( x ; - x)'UU'(Xy - x)] 

i=i ;=i 

= £tr[UU'(xy-x)(xy-x)'] 

i=i 

= (n - 1) tr[UU'S] = («-!) trfU'SU] (8A-4) 
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That is, the best choice for U maximizes the sum of the diagonal elements of U'SU 
From (8-19), selecting u, to maximize uJSu|, the first diagonal element of U'SU, gives 
U{ = © 1 . For U 2 perpendicular to ej, U 2 S 112 is maximized by e 2 . [See (2-52).] Continuing 
we find that U = [e lf c 2 , ■.., e r ] = E r and A' = E r Er[ x i - x, x 2 - x,..., x n - x], as 
asserted. „ 

With this choice the zth diagonal element of U'SU is e'Se,- = ej(A/e,-) = A, so 


tr [U'SU] = Aj + A 2 + ■ * • + A r . Also, 2 (x y - x)'(x y - x) = tr ^ ( x y “ x ) ( x y - x)‘ 

7=1 L 7=i 


= (n - 1) tr(S) = (« “ l)(Ai + A 2 + *’• + A^). Let U = U in (8A-3), and the 
error bound follows. ■ 


The p-Dimensional Geometrical Interpretation 

The geometrical interpretations involve the determination of best approximating 
planes to the p-dimensional scatter plot. The plane through the origin, determined 
by Ui, u 2 , • ■ , u r , consists of all points x with 

x = b]Ui + b 2 u 2 + + b,.u r = Ub, for some b 

This plane, translated to pass through a, becomes a + Ub for some b. 

We want to select the /--dimensional plane a + Ub that minimizes the sum of 

n 

squared distances ^ dj between the observations x y and the plane. If x y is approxi- 

y=i 

n 

mated by a + Ub y with ^ b y = 0, 5 then 
7=1 

2(x y -a-Ub y )'(x y -a-Ub y ) 

7 = 1 

n 

= 2 ( x y ~ * “ Ub y + x — a)'(x y - x - Ub y + x - a) 

/■I 

= 2 (*> “ x “ Ub y )'(x y - x - Uby) + «( x - a)'( x - a) 

7=1 

2= 2 (*y - x - E r Er(x; - x))'(xy - x - E r E' r (Xy - x)) 

7 = 1 

by Result 8A.1, since [Ubj,..., Ub„] = A' has rank (A) ^ r. The lower bound is 
reached by taking a = x, so the plane passes through the sample mean. This plane is 
determined by £ 1 , e 2 ,...,e r . The coefficients of e k are e*(x y - x) = y jk , the fcth 
sample principal component evaluated at theyth observation. 

The approximating plane interpretation of sample principal components is 
illustrated in Figure 8.10. 

An alternative interpretation can be given. The investigator places a plane 
through x and moves it about to obtain the largest spread among the shadows of the 

5 If 2 b; = nb * 0, use a + Ub y = (a + Ub) + U(b y - b) = a* + Ub*. 

j-i 
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observation;;. From (8A-2), the projection of the deviation x, - x on the plane Ub is 
^7 \Xj x )* Now, v — 0 and the sum of the squared lengths of the projection 

» n 

2 v / v / = 2 (Xy “ x)'UU'(xy - x) = (n - 1) tr[U'SU] 

is maximized by U = E. Also, since v = 0, 

(” ~ 1)S» = 2 (V/ - V)(Vy - - 2 VyVy 

y-i 

and this plane also maximizes the total variance 


tr (S») 



tr 




tr 



Let us now consider, by columns, the approximation of the mean-centered data 
matrix by A. For r = 1, the fth column [x u - x„ x v - x„, - 3c,]' is approxi- 

mated by a multiple c,b' of a fixed vector b' = [b,, b 2 ,..., b,,]. The square of the 
length of the error of approximation is 

= 2 (•*■;■ - *i ~ c ibj) 2 

7=1 

Considering to be of rank one, we conclude from Result 8A.1 that 
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Exercises 

s.i. 


8 . 2 . 





(a) Principal component of S (b) Principal component of R 

Figure 8.11 The first sample principal component, y, , minimizes the 
sum of the squares of the distances, L] , from the deviation vectors, 
d'i = [xi f - x h x;i, — x h - x ni — x,], to a line. 


minimizes the sum of squared lengths 2 ^ ■ ^ lal , * ie * 5es * direction is determined 

by the vector of values of the first principal component. This is illustrated in 
Figure 8.11(a). Note that the longer deviation vectors (the larger s„’s) have the most 

P 

influence on the minimization of ^ L t . 

If the variables are first standardized, the resulting vector [(x,, - x,)/Vs^, 
(x 2 , - Xi)/V%„ ■.., (x ni - 3E,)/Vsi/1 has length n - 1 for all variables and each 


vector exerts equal influence on the choice of direction. [See Figure 8.11(b).] 

In either case, the vector b is moved around in n-space to minimize the sum of 

the squares of the distances ^ L ). In the former case L? is the squared distance 

between [x 1( - x h x 2i -*,,.!•!x„, - x,]' and its projection on the line determined 
by b. The second principal component minimizes the same quantity among all 
vectors perpendicular to the first choice. 


Determine the population principal components Yi and Y 2 for the covariance matrix 

x = P 2 1 

L 2 2 J 

Also, calculate the proportion of the total population variance explained by the first 
principal component. 

Convert the covariance matrix in Exercise 8.1 to a correlation matrix p. 

(a) Determine the principal components Yi and Y 2 from p and compute the proportion 
of total population variance explained by Y y . 
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(b) Compare the components calculated in Part a with those obtained in Exercise 8.1. 
Are they the same? Should they be? 

(c) Compute the correlations py ltZl , Py u z 2 > and Py 2 ,z r 

8.3. Let 

2 0 (T 

0 4 0 
0 0 4_ 

Determine the principal components Yj, Y 2 , and Y 3 . What can you say about the eigen¬ 
vectors (and principal components) associated with eigenvalues that are not distinct? 

8.4. Find the principal components and the proportion of the total population variance 
explained by each when the covariance matrix is 

cr 2 cr 2 p 0 

2 = cr 2 p cr 2 cr 2 p , 

0 cr 2 p cr 2 

8.5. (a) Find the eigenvalues of the correlation matrix 

1 p p 

P = P 1 P 
_P P 

Are your results consistent with (8-16) and (8-17)? 

(b) Verify the eigenvalue-eigenvector pairs for the p X p matrix p given in (8-15). 

8 . 6 . Data on = sales and x 2 = profits for the 10 largest companies in the world were 
listed in Exercise 1.4 of Chapter 1. 

From Example 4.12 


155.60* 

1" 7476.45 

303.62 ~ 

14.70_ ’ 

|_ 303.62 

26.19_ 


(a) Determine the sample principal components and their variances for these data. (You 
may need the quadratic formula to solve for the eigenvalues of S.) 

(b) Find the proportion of the total sample variance explained by y x . 

(c) Sketch the constant density ellipse (x — x)'S -1 (x - x) =1.4, and indicate the 
principal components y Y and yi on your graph. 

(d) Compute the correlation coefficients rp Xk >k = 1,2. What interpretation, if any, can 
you give to the first principal componefit? 

8.7. Convert the covariance matrix S in Exercise 8.6 to a sample correlation matrix R. 

(a) Find the sample principal components y lt y 2 and their variances. 

(b) Compute the proportion of the total sample variance explained by j^. 

(c) Compute the correlation coefficients rp 1%tk , k = 1,2. Interpret y Y . 

(d) Compare the components obtained in Part a with those obtained in Exercise 8.6(a). 
Given the original data displayed in Exercise 1.4, do you feel that it is better to 
determine principal components from the sample covariance matrix or sample 
correlation matrix? Explain. 
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8 . 8 . Use the results in Example 8.5. ^ 

(a) Compute the correlations r y . lk for / = 1,2 and k - 1,2,... ,5. Do these correla -M 
tions reinforce the interpretations given to the first two components. Explain. 

(b) Test the hypothesis 


Hq’- p — Pq — 


1 p P P P 

P 1 P P P 

p p 1 p p 

P P P 1 P 

_P P P P 


versus 


H,:p * p 0 

at the 5% level of significance. List any assumptions required in carrying out this test. 

8.9. (A test that all variables are independent.) 

(a) Consider that the normal theory likelihood ratio test of H 0 : X is the diagonal matrix 


<7n 0 

0 < 7 2 2 

0 0 


0 ' 
0 

a pp. 


, C7jj > 0 


Show that the test is as follows: Reject H 0 if 

I S I "/ 2 

A =- ! f J —= |R| nl2 <c 

Usf 

i= 1 

For a large sample size, -2 In A is approximately x\( P - 1)/2- Bartlett [3] suggests that 
the test statistic -2[1 - (2 p + ll)/6n]lnA be used in place of —2 In A. This 
results in an improved chi-square approximation. The large sample a critical point is 
Xp(p-i)/2(<*)- Note that testing 2 = S 0 ls the same as testing p = I. 

(b) Show that the likelihood ratio test of H 0 : 2 = <r 2 1 rejects H 0 if 


isr /2 


(tT(S)/p) np > 2 


fii, 


\n}2 




geometric mean A 
.arithmetic mean A 


[l n P / 2 
- < c 

U 


For a large sample size, Bartlett [3] suggests that 

-2[1 - (2 p 2 + p + 2)/6 pn] In A 

is approximately *( p+2 )(p-i)/2- Thus, ^ lar 8 e sam P le a crltlcal 
A( P + 2 )(p-i)/ 2 ( a )* This test is called a sphericity test , because the constant density 

contours are spheres when X = <7 2 I. 



Hint: 

(a) maxL(/*, X) is given by (5-10), and max X 0 ) is the product of the univariate 


likelihoods, max(27r) n/2 cr u n / 2 ex p 


(*/,—Mi) 2 /2<7f# ■ Hence Pi = n 1 ^ x ji 




and an ~ (1 /n) ^ ( x ji ~ x i) 2 ' The divisor n cancels in A, so S may be used. 


/ 


(b) Verify a 2 = | ^ (x fl - x x ) + ♦•• + X ( x jp ~ x p ) I / n P under H 0 . Again, 
-/• 1 

the divisors n cancel in the statistic, so S may be used. Use Result 5.2 to calculate the 
chi-square degrees of freedom. 


The following exercises require the use of a computer. 

8 . 10. The weekly rates of return for five stocks listed on the New York Stock Exchange are given 

in Table 8.4. (See the stock-price data on the following website: www.prenhall.com/statistics.) 

(a) Construct the sample covariance matrix S, and find the sample principal components 
in (8-20). (Note that the sample mean vector x is displayed in Example 8.5.) 

(b) Determine the proportion of the total sample variance explained by the first three 
principal components. Interpret these components. 

(c) Construct Bonferroni simultaneous 90% confidence intervals for the variances 
Ai, A 2 , and A 3 of the first three population components Y \, Y 2 , and Y 3 . 

(d) Given the results in Parts a-c, do you feel that the stock rates-of-return data can be 
summarized in fewer than five dimensions? Explain. 


Table 8.4 Stock-Price Data (Weekly Rate Of Return) 


Week 

JP 

Morgan 

Citibank 

Wells 

Fargo 

Royal 
Dutch Shell 

Exxon 

Mobil 

1 

0.01303 

-0.00784 

-0.00319 

-0.04477 

0.00522 

2 

0.00849 

0.01669 

-0.00621 

0.01196 

0.01349 

3 

-0.01792 

-0.00864 

0.01004 

0 

-0.00614 

4 

0.02156 

-0.00349 

0.01744 

-0.02859 

-0.00695 

5 

0.01082 

0.00372 

-0.01013 

0.02919 

0.04098 

6 

0.01017 

-0.01220 

-0.00838 

0.01371 

0.00299 

7 

0.01113 

0.02800 

0.00807 

0.03054 

0.00323 

8 

0.04848 

-0.00515 

0.01825 

0.00633 

0.00768 

9 

-0.03449 

-0.01380 

-0.00805 

-0.02990 

-0.01081 

10 

-0.00466 

0.02099 

-0.00608 

-0.02039 

-0.01267 

94 

0.03732 

0.03593 

0.02528 

0.05819 

0.01697 

95 

0.02380 

0.00311 

-0.00688 

0.01225 

0.02817 

96 

0.02568 

0.05253 

0.04070 

-0.03166 

-0.01885 

97 

-0.00606 

0.00863 

0.00584 

0.04456 

0.03059 

98 

0.02174 

0.02296 

0.02920 

0.00844 

0.03193 

99 

0.00337 

-0.01531 

-0.02382 

-0.00167 

-0.01723 

100 

0.00336 

0.00290 

-0.00305 

-0.00122 

-0.00970 

101 

0.01701 

0.00951 

0.01820 

-0.01618 

-0.00756 

102 

0.01039 

-0.00266 

0.00443 

-0.00248 

-0.01645 

103 

-0.01279 

-0.01437 

-0.01874 

-0.00498 

-0.01637 
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8 11 Consider the census-tract data listed in Table 8.5. Suppose the observations on 
8.11. Consul were recorded in ten thousand*rather than hundred thousand* 

afdollar*that i* multiply all the numbers listed in the sixth column of the table by 10 . 
tat rvmstnict the sample covariance matrix S for the census-tract data when 
y = median value home is recorded in ten thousands of dollar* (Note that this - 
covariance matrix can be obtained from the covariance matrix given in Example 83 
Z multiplying the off-diagonal elements in the fifth column and row by 10 and the 
diagonal element s 5S by 100. Why?) 

(b) Obtain the eigenvalue-eigenvector pairs and the first two sample principal compo¬ 
nents for the covariance matrix in Part a. 

trl Comoute the proportion of total variance explained by the first two principal 
( ) components obtained in Part b. Calculate the correlation “efficient* r y and 
interpret these components if possible. Compare your results with the results in' 
Example 8.3. What can you say about the effects of this change in scale on the 
principal components? 

8 12 Consider the air-pollution data listed in Table 1.5. Your job is to summarize these data in 
2 than p = 7 dimensions if possible. Conduct a pnncipal component ana ys.s of the 
fewer man p riance matrix S and the correlation matrix R. What have you 

learned? Does it make any difference which matrix is chosen for analysis? Can the data be 
summarized in three or fewer dimensions? Can you interpret the principal components? 


Table 8.5 Census-tract Data 


Tract 


1 

2 

3 

4 

5 

- 6 

7 

8 
9 

10 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 


Total 

population 

(thousands) 


2.67 

2.25 
3.12 

5.14 
5.54 
5.04 

3.14 

2.43 
5.38 
7.34 

7.25 

5.44 
5.83 
3.74 
9.21 

2.14 
6.62 
4.24 
4.72 
6.48 


Professional 

degree 

(percent) 


5.71 

4.37 

10.27 

7.44 

9.25 
4.84 

4.82 
2.40 

4.30 
2.73 

1.16 

2.93 
4.47 

2.26 
2.36 

6.30 
4.79 

5.82 
4.71 

4.93 


Employed 
age over 16 
(percent) 


69.02 

72.98 

64.94 
71.29 

74.94 
53.61 
67.00 
67.20 
83.03 
72.60 

78.52 

73.59 

77.33 

79.70 

74.58 

86.54 

78.84 

71.39 

78.01 

74.23 


Government 

employment 

(percent) 


Median 
home value 
($ 100 , 000 ) 


-“ 7 « nHiacent census tracts are likely to be correlated. That is, these 61 observations may not 

corlsti tute a'random sample. Complete data set available atwww.prenhall.com/statistics.___ 
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8.13. In the radiotherapy data listed in Table 1.7 (see also the radiotherapy data on the 
website www.prenhall.com/statistics), the n = 98 observations on p - 6 variables rep¬ 
resent patients’ reactions to radiotherapy. 

(a) Obtain the covariance and correlation matrices S and R for these data. 

(b) Pick one of the matrices S or R (justify your choice), and determine the eigenval¬ 
ues and eigenvectors. Prepare a table showing, in decreasing order of size,, the per¬ 
cent that each eigenvalue contributes to the total sample variance. 

(c) Given the results in Part b, decide on the number of important sample principal 
components. Is it possible to summarize the radiotherapy data with a single reaction- 
index component? Explain, 

(d) Prepare a table of the correlation coefficients between each principal component 
you decide to retain and the original variables. If possible, interpret the components. 

8.14. Perform a principal component analysis using the sample covariance matrix of the 
sweat data given in Example 5.2. Construct a Q~Q plot for each of the important 
principal components. Are there any suspect observations? Explain. 

8.15. The four sample standard deviations for the postbirth weights discussed in Example 8.6 
are 

= 32.9909, Vs^ = 33.5918, Vs 7 = 36.5534, and Vs 7 = 37.3517 

Use these and the correlations given in Example 8.6 to construct the sample covariance 
matrix S. Perform a principal component analysis using S. 

8.16. Over a period of five years in the 1990s, yearly samples of fishermen on 28 lakes in 
Wisconsin were asked to report the time they spent fishing and how many of each 
type of game fish they caught. Their responses were then converted to a catch rate per 
hour for 

x t = Bluegill x 2 = Black crappie x 3 = Smallmouth bass 

x 4 = Largemouth bass x 5 = Walleye x 6 = Northern pike 

The estimated correlation matrix (courtesy of Jodi Barnet) 


1 

.4919 

.4919 

1 

.2635 

.3127 

.4653 

.3506 

-.2277 

-.1917 

.0652 

.2045 


.2636 

.4653 

.3127 

.3506 

1 

.4108 

.4108 

1 

.0647 

-.2249 

.2493 

.2293 


-.2277 

.0652 

-.1917 

.2045 

.0647 

.2493 

-.2249 

.2293 

1 

-.2144 

-.2144 

1 


is based on a sample of about 120. (There were a few missing values.) 

Fish caught by the same fisherman live alongside of each other, so the data should 
provide some evidence on how the fish group. The first four fish belong to the centrar- 
chids, the most plentiful family. The walleye is the most popular fish to eat. 

(a) Comment on the pattern of correlation within the centrarchid family Xi through x 4 . 
Does the walleye appear to group with the other fish? 

(b) Perform a principal component analysis using only x r through x 4 . Interpret your 
results. 

(c) Perform a principal component analysis using all six variables. Interpret your results. 
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8.17. Using the data on bone mineral content in Table 1.8, perform a principal component 
analysis of S. 

8.18. The data on national track records for women are listed in Table 1.9. 

(a) Obtain the sample correlation matrix R for these data, and determine its eigenvalues : 
and eigenvectors. 

(b) Determine the first two principal components for the standardized variables. Pre¬ 
pare a table showing the correlations of the standardized variables with the compos 
nents, and the cumulative percentage of the total (standardized) sample variance 
explained by the two components. 

(c) Interpret the two principal components obtained in Part b. (Note that the first 
component is essentially a normalized unit vector and might measure the athlet¬ 
ic excellence of a given nation. The second component might measure the rela¬ 
tive strength of a nation at the various running distances.) 

fd) Rank the nations based on their score on the first principal component Does this 
ranking correspond with your inituitive notion of athletic excellence for the various 
countries? 

8.19. Refer to Exercise 8.18. Convert the national track records for women in Table L9 to 
speeds measured in meters per second. Notice that the records for 800 m, 1500 m, 
3000 m and the marathon are given in minutes. The marathon is 26.2 miles, or 
42 195 meters, long. Perform a principal components analysis using the covariance 
matrix S of the speed data. Compare the results with the results in Exercise 818. Do 
your interpretations of the components differ? If the nations are ranked on the basis of 
their spore on the first principal component, does the subsequent ranking differ from 
that in Exercise 8.18? Which analysis do you prefer? Why? 

8 20. The data on national track records for men are listed in Table 8.6. (See also the data 
on national track records for men on the website www.prenhall.com/statistics) Repeat 
the principal component analysis outlined in Exercise 8.18 for the men. Are the results 
consistent with those obtained from the women’s data? 

8.21. Refer to Exercise 8.20. Convert the national track records for me ' l i " Tab '® 8 '^ 

measured in meters per second. Notice that the records for 800 m, 1500 m, 5000 m, 
10 000 m and the marathon are given in minutes. The marathon is 26.2 miles, or 
42 195 meters, long. Perform a principal component analysis using the covariance matrix 
S of the speed data. Compare the results with the results in Exercise 8.20. Which analy 
do you prefer? Why? 

8 22 Consider the data on bulls in Table 1.10. Utilizing the seven variables YrHgt, FtFrBody 
‘ 2 ' PrctFFB Frame, BkFat, SaleHl, and SaleWt, perform a principal component anal^is 
using the covariance matrix S and the correlation matrix R. Your analysis should include 
the following: . 

(a) Determine the appropriate number of components to effectively summarize the 
sample variability. Construct a scree plot to aid your determination. 

(b) Interpret the sample principal components. 

(c) Do you think it is possible to develop a “body size" or “body configuration” index 
from the data on the seven variables above? Explain. 

(d) Using the values for the first two principal components, plot the datai in a, two- 
dimensional space with y, along the vertical axis and y 2 along ^ honzonfol a 
Can you distinguish groups representing the three breeds of cattle. Are th Y 
outliers? 

(e) Construct a Q-Q plot using the first principal component. Interpret the plot. 
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Table 8.6 National lYack Records for Men 


Country 

100 m 

(s) 

200 m 

(s) 

400 m 

(s) 

800 m 
(min) 

1500 m 
(min) 

5000 m 
(min) 

10,000 m 
(min) 

Marathon 

(min) 

Argentina 

10.23 

20.37 

46.18 

1.77 

3.68 

13.33 

27.65 

129.57 

Australia 

9.93 

20.06 

44.38 

1.74 

3.53 

12.93 

27.53 

127.51 

Austria 

10.15 

20.45 

45.80 

1.77 

3.58 

13.26 

27.72 

132.22 

Belgium 

10.14 

20.19 

45.02 

1.73 

3.57 

12.83 

26.87 

127.20 

Bermuda 

10.27 

20.30 

45.26 

1.79 

3.70 

14.64 

30.49 

14637 

Brazil 

10.00 

19.89 

44.29 

1.70 

3.57 

13.48 

28.13 

126.05 

Canada 

9.84 

20.17 

44.72 

1.75 

3.53 

13.23 

27.60 

130.09 

Chile 

10.10 

20.15 

45.92 

1.76 

3.65 

13.39 

28.09 

132.19 

China 

10.17 

20.42 

45.25 

1.77 

3.61 

13.42 

28.17 

129.18 

Columbia 

10.29 

20.85 

45.84 

1.80 

3.72 

13.49 

27.88 

131.17 

Cook Islands 

10.97 

22.46 

51.40 

1.94 

4.24 

16.70 

35.38 

171.26 

Costa Rica 

1032 

20.96 

46.42 

1.87 

3.84 

13.75 

28.81 

133.23 

Czech Republic 

10.24 

20.61 

45.77 

1.75 

3.58 

13.42 

27.80 

131.57 

Denmark 

10.29 

20.52 

45.89 

1.69 

3.52 

13.42 

27.91 

129.43 

Dom in icanRepubl ic 

10.16 

20.65 

44.90 

1.81 

3.73 

14.31 

30.43 

146.00 

Finland 

10.21 

20.47 

45.49 

1.74 

3.61 

13.27 

27.52 

131.15 

France 

10.02 

20.16 

44.64 

1.72 

3.48 

12.98 

2738 

126.36 

Germany 

10.06 

20.23 

44.33 

1.73 

3.53 

12.91 

2736 

128.47 

Great Britain 

9.87 

19.94 

44.36 

1.70 

3.49 

13.01 

2730 

127.13 

Greece 

10.11 

19.85 

45.57 

1.75 

3.61 

13.48 

28.12 

132.04 

Guatemala 

10.32 

21.09 

48.44 

1.82 

3.74 

13.98 

' 29.34 

132.53 

Hungary 

10.08 

20.11 

45.43 

1.76 

3.59 

13.45 

28.03 

132.10 

India 

10.33 

20.73 

45.48 

1.76 

3.63 

13.50 

28.81 

132.00 

Indonesia 

10.20 

20.93 

46.37 

1.83 

3.77 

14.21 

29.65 

139.18 

Ireland 

10.35 

20.54 

45.58 

1.75 

3.56 

13.07 

27.78 

129.15 

Israel 

10.20 

20.89 

46.59 

1.80 

3.70 

13.66 

28.72 

134.21 

Italy 

10.01 

19.72 

45.26 

1.73 

3.35 

13.09 

27.28 

127.29 

Japan 

10.00 

20.03 

44.78 

1.77 

3.62 

13.22 

27.58 

126.16 

Kenya 

10.28 

20.43 

44.18 

1.70 

3.44 

12.66 

26.46 

124.55 

Korea, South 

10.34 

20.41 

45.37 

1.74 

3.64 

13.84 

28.51 

127.20 

Korea, North 

10.60 

21.23 

46.95 

1.82 

3.77 

13.90 

28.45 

129.26 

Luxembourg 

10.41 

20.77 

47.90 

1.76 

3.67 

13.64 

28.77 

134.03 

Malaysia 

10.30 

20.92 

46.41 

1.79 

3.76 

14.11 

29.50 

149.27 

Mauritius 

10.13 

20.06 

44.69 

1.80 

3.83 

14.15 

29.84 

143.07 

Mexico 

10.21 

20.40 

44.31 

1.78 

3.63 

13.13 

27.14 

127.19 

Myanmar(Burma) 

10.64 

21.52 

48.63 

1.80 

3.80 

14.19 

29.62 

139.57 

Netherlands 

10.19 

20.19 

45.68 

1.73 

3.55 

13.22 

27.44 

12831 

New Zealand 

10.11 

20.42 

46.09 

1.74 

3.54 

13.21 

27.70 

128.59 

Norway 

10.08 

20.17 

46.11 

1.71 

3.62 

13.11 

27.54 

130.17 

Papua New Guinea 

10.40 

21.18 

46.77 

1.80 

4.00 

14.72 

3136 

148.13 

Philippines 

10.57 

21.43 

45.57 

1.80 

3.82 

13.97 

29.04 

138.44 

Poland 

10.00 

19.98 

44.62 

1.72 

3.59 

13.29 

27.89 

129.23 

Portugal 

9.86 

20.12 

46.11 

1.75 

3.50 

13.05 

27.21 

126.36 

Romania 

10.21 

20.75 

45.77 

1.76 

3.57 

13.25. 

27.67 

132.30 

Russia 

10:11 

20.23 

44.60 

1.71 

3.54 

13.20 

27.90 

129.16 

Samoa 

10.78 

21.86 

49.98 

1.94 

4.01 

16.28 

34.71 

161.50 

Singapore 

10.37 

21.14 

47.60 

1.84 

3.86 

14.96 

3132 

144.22 

Spain 

10.17 

20.59 

44.96 

1.73 

3.48 

13.04 

27.24 

127.23 

Sweden 

10.18 

20.43 

45.54 

1.76 

3.61 

13.29 

27.93 

130.38 

Switzerland 

10.16 

20.41 

44.99 

1.71 

3.53 

13.13 

27.90 

129.56 

Taiwan 

10.36 

20.81 

46.72 

1.79 

3.77 

13.91 

29.20 

13435 

Thailand 

10.23 

20.69 

46.05 

1.81 

3.77 

14.25 

29.67 

13933 

Tlirkey 

10.38 

21.04 

46.63 

1.78 

3.59 

13.45 

28.33 

130.25 

US. A. 

9.78 

19.32 

43.18 

1.71 

3.46 

12.97 

27.23 

12538 


Source: 1AAF/ATES TYack and Field Statistics Handbook for the Helsinki 2005 Olympics. Courtesy of Ottavio Castellini, 
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8.23. A naturalist for the Alaska Fish and Game Department studies grizzly bears with the 
goal of maintaining a healthy population. Measurements on n = 61 bears provided the 
following summary statistics: 


Variable 

Weight 

(kg) 

Body 

length 

(cm) 

Neck 

(cm) 

Girth 

(cm) 

Head 

length 

(cm) 

Head 

width 

(cm) 

Sample 
mean x 

95.52 

164.38 

55.69 

93.39 

17.98 

31.13 


Covariance matrix 


"3266.46 1343.97 731.54 1175.50 162.68 238.37" 


1343.97 

721.91 

324.25 

537.35 

80.17 

117.73 

731.54 

324.25 

179.28 

281.17 

39.15 

56.80 

1175.50 

537.35 

281.17 

474.98 

63.73 

94.85 

162.68 

80.17 

39.15 

63.73 

9.95 

13.88 

238.37 

117.73 

56.80 

94.85 

13.88 

21.26 


(a) Perform a principal component analysis using the covariance matrix. Can the data 
be effectively summarized in fewer than six dimensions? 

(b) Perform a principal component analysis using the correlation matrix. 

(c) Comment on the similarities and differences between the two analyses. 

8.24- Refer to Example 8.10 and the data in Table 5.8, page 240. Add the variable x 6 = regular 

overtime hours whose values are (read across) 

6187 7336 6988 6964 8425 6778 5922 7307 

7679 8259 10954 9353 6291 4969 4825 6019 

and redo Example 8.10. 

8.25. Refer to the police overtime hours data in Example 8.10. Construct an alternate control 
chart, based on the sum of squares dyj , to monitor the unexplained variation in the orig 
inal observations summarized by the additional principal components. 

8.26. Consider the psychological profile data in Table 4.6. Using the five variables, Indep, Supp, 
Benev, Conform and Leader, performs a principal component analysis using the covari 
ance matrix S and the correlation matrix R. Your analysis should include the following: 

(a) Determine the appropriate number of components to effectively summarize the 
variability. Construct a scree plot to aid in your determination. 

(b) Interpret the sample principal components. 

(c) Using the values for the first two principal components, plot the data in a two 
dimensional space with y\ along the vertical axis and y 2 along the horizontal axis. 
Can you distinguish groups representing the two socioeconomic levels and/or the 
two genders? Are there any outliers? 

(d) Construct a 95% confidence interval for Aj, the variance of the first population 
principal component from the covariance matrix. 

8.27. The pulp and paper properties data is given in Table 7.7. Using the four paper variables, 
BL (breaking length), EM (elastic modulus), SF (Stress at failure) and BS (burst 
strength), perform a principal component analysis using the covariance matrix S and the 
correlation matrix R. Your analysis should include the following: 

(a) Determine the appropriate number of components to effectively summarize the 

variability. Construct a scree plot to aid in your determination. 



Exercises 479 


(b) Interpret the sample principal components. 

(c) Do you think it it is possible to develop a “paper strength” index that effectively con¬ 
tains the information in the four paper variables? Explain. 

(d) Using the values for the first two principal components, plot the data in a two- 

1003 S ^f 1 Ce Wlth ^ a ^ on 8 vertical axis and y 2 along the horizontal axis. 
Identify any outliers in this data set. 

S r re c ° llect ^ d as P art of a stud y t0 assess options for enhancing food secu- 

sustaina ^ le use of na tural resources in the Sikasso region of Mali (West 
Ainca). A total of n - 76 farmers were surveyed and observations on the nine variables 

x, = Family (total number of individuals in household) 
x 2 = DistRd (distance in kilometers to nearest passable road) 

* 3 = Cotton (hectares of cotton planted in year 2000) 
x 4 = Maize (hectares of maize planted in year 2000) 

* 5 = Sorg (hectares of sorghum planted in year 2000) 
x 6 = Millet (hectares of millet planted in year 2000) 
x n = Bull (total number of bullocks or draft animals) 

*8 = Cattle (total); *9 = Goats (total) 

were recorded. The data are listed in Table 8.7 and on the website www.prenhall.com/statistics 
(a) Construct two-dimensional scatterplots of Family versus DistRd, and DistRd versus 
Cattle. Remove any obvious outliers from the data set. 
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(b) Perform a principal component analysis using the correlation matrix R. Determine 
the number of components to effectively summarize the variability. Use the propor¬ 
tion of variation explained and a scree plot to aid in your determination. 

(c) Interpret the first five principal components. Can you identify, for example, a “farm 

size” component? A, perhaps, “goats and distance to road” component? _ 

8.29. Refer to Exercise 5.28. Using the covariance matrix S for the first 30 cases of car body 
assembly data, obtain the sample principal components. 

(a) Construct a 95% ellipse format chart using the first two principal components y x andT 
y 2 - Identify the car locations that appear to be out of control. 

(b) Construct an alternative control chart, based on the sum of squares d}j to monitor 
the variation in the original observations summarized by the remaining four princi-"" 
pal components. Interpret this chart. 
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Factor Analysis and Inference 
for Structured Covariance 
Matrices 

9.1 Introduction 

Factor analysis has provoked rather turbulent controversy throughout its history. Its 
modern beginnings lie in the early-20th-century attempts of Karl Pearson, Charles 
Spearman, and others to define and measure intelligence. Because of this early 
association with constructs such as intelligence, factor analysis was nurtured and 
developed primarily by scientists interested in psychometrics. Arguments over the 
psychological interpretations of several early studies and the lack of powerful com¬ 
puting facilities impeded its initial development as a statistical method. The advent 
of high-speed computers has generated a renewed interest in the theoretical and 
computational aspects of factor analysis. Most of the original techniques have been 
abandoned and early controversies resolved in the wake of recent developments. It 
is still true, however, that each application of the technique must be examined on its 
own merits to determine its success. 

The essential purpose of factor analysis is to describe, if possible, the covariance 
relationships among many variables in terms of a few underlying, but unobservable, 
random quantities called factors . Basically, the factor model is motivated by the 
following argument. Suppose variables can be grouped by their correlations. That is, 
suppose all variables within a particular group are highly correlated among them¬ 
selves, but have relatively small correlations with variables in a different group. Then 
it is conceivable that each group of variables represents a single underlying construct, 
or factor, that is responsible for the observed correlations. For example, correlations 
from the group of test scores in classics, French, English, mathematics, and music 
collected by Spearman suggested an underlying “intelligence” factor. A second group 
of variables, representing physical-fitness scores, if available, might correspond to 
another factor. It is this type of structure that factor analysis seeks to confirm. 
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Factor analysis can be considered an extension of principal component analysis. 
Both can be viewed as attempts to approximate the covariance matrix X. However 
the approximation based on the factor analysis model is more elaborate. The- 
primary question in factor analysis is whether the data are consistent with a 
prescribed structure. 


The Orthogonal Factor Model 

The observable random vector X, with p components, has mean p and covariance ' 
matrix X. The factor model postulates that X is linearly dependent upon a few un¬ 
observable random variables Fy , F 2 ,..., F m , called common factors , and p addition¬ 
al sources of variation ey , e 2 » • • •»called errors or, sometimes, specific factors} In 
particular, the factor analysis model is 

X\ — p\ — Z\\F\ + £ 12 F 2 + • • * + £\ m F m + Ey 

Xl - P '2 = £l\F\ + ^ 22^2 +- 1 - £2 m Fm + e 2 ^ 

X p ~ p p - £ p \F l + £ p 2 F 2 + + £ pm F m + e p 


or, in matrix notation, 


X — /* = L F + e 

(pxi) (pXm)(mXl) (pX 1) 


The coefficient £^ is called the loading of the ith variable on the jth factor, so the matrix 
L is the matrix of factor loadings. Note that the ith specific factor e* is associated only 
with the ith response X { , The p deviations Xy - p\, X 2 - P 2 ,...» X p - p p are 
expressed in terms of p + m random variables Fy, F 2 ,... , F m , , e 2 ,..., s p which are 

unobservable. This distinguishes the factor model of (9-2) from the multivariate regres¬ 
sion model in (7-23), in which the independent variables [whose position is occupied by 
Fin (9-2)] can be observed. 

With so many unobservable quantities, a direct verification of the factor model 
from observations on Xy , X 2 ,..., X p is hopeless. However, with some additional 
assumptions about the random vectors F and e, the model in (9-2) implies certain 
covariance relationships, which can be checked. 

We assume that 


E{ F) = 0 , 

v (mxl) 


E(£) = 0 , 
V (pxi) 


Cov(F) = £TFF'] = I 

L (mXm) 


Cov(e) = E\e£'\ = ¥ 
(pxp) 


ipi 0 

0 ip 2 


1 As Maxwell [12] points out, in many investigations the e t tend to be combinations of measurement 
error and factors that are uniquely associated with the individual variables. 
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and that F and e are independent, so 

Cov(e,F) = E(eF') = 0 

(pXm) 

TTiese assumptions and the relation in (9-2) constitute the orthogonal factor model . 2 

Orthogonal Factor Model with m Common Factors 

. X = p + L F + e 

(p*l) (pX 1) [pXm)(mX 1) (pxi) 

Pi = mean of variable i 

e i ~ *th specific factor (9,4) 

Fj — ;th common factor 

(ij = loading of the ith variable on the ;th factor 

The unobservable random vectors F and c satisfy the following conditions: 

F and e are independent 
E( F) = 0, Cov(F) = I 

E(e) — 0, Cov(e) = ¥, where ¥ is a diagonal matrix 

model'fn (9-4) 8 ° nal m ° del implies a covariance structure for X. From the 

(X ~ m)(X - n)' = (LF + e)(LF + e)' 

= (LF + e)((LF)' + e') 

= LF(LF)' + e(LF)' + LFe + ee' 

so that 

X = Cov(X) = E(X - fi)(X - M y 

= L£(FF')L' + E(eF') L + LE(Fe') + E(ee') 

= LL' + ¥ 

according to (9-3). Also by independence, Cov (e, F) = E(e F') = 0 

Covrx Ff b - C X m ° de L in (9 ‘ 4) ’ (x ~ M)F ’ = ( LF+ «)F' = LFF' + «F'. 

Cov (X, F) - E(X - fi)F = L£(FF') + E(sF’) = L. 


tbe faC ' or , s F to be correlated so that Cov(F) is nor diagonal p- 
“(stXr* 1 PreS6ntS SOme additional estimation difficulties ar 
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The equality 

19 30 2 12"] 4 1" 

30 57 5 23 7 2 [4 7 

2 5 38 47 “ -1 6 |_1 2 

.12 23 47 68J [_ 1 8J 

or 

2 = LL + ¥ 

may be verified by matrix algebra. Therefore, 2 has the structure produced by an 
m = 2 orthogonal factor model. Since 


("2 0 0 O' 

-1 ll 0 4 0 0 

6 8j + 0 0 1 0 

[_0 0 0 3_ 



<Ai o 0 0] [200 0' 

^ = o ip 2 0 0 _ 0 4 0 0 

o 0 ip 3 0 0 0 1 0 

_ 0 0 0 ip 4 j [_0 0 0 3_ 


the communality of X x is, from (9-6), 

h \ = ^ii + t\i = 4 2 + l 2 = 17 
and the variance of X\ can be decomposed as 

°Ti = (^ii + ^12) + <Ai — h\ + ip l 


or 



variance = communality + specific 

variance 


17 + 2 


A similar breakdown occurs for the other variables. 


The factor model assumes that the p + p(p - 1 )/2 = p(p + l )/2 variances and 
covariances for X can be reproduced from the pm factor loadings €/y and the p specif¬ 
ic variances <//,-. When m = p, any covariance matrix 2 can be reproduced exactly as 
LL' [see (9-11)], so ¥ can be the zero matrix. However, it is when m is small relative 
to p that factor analysis is most useful. In this case, the factor model provides a “ r ’ 
pie” explanation of the covariation in X with fewer parameters than the p(p 
parameters in X. For example, if X contains p = 12 variables, and the factr 
(9-4) with m - 2 is appropriate, then the p(p + l)/2 = 12(13)/2 = n ' 

2 are described in terms of the mp + p = 12(2) + 12 = 36param 
the factor model. 
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Unfortunately for the factor analyst, most covariance matrices cannot be fac- 
tored as LL' + where the number of factors m is much less than p.The following 

example demonstrates one of the problems that can arise when attempting to deter- 
mine the parameters and 0, from the variances and covariances of the observable -Z 

variables. 

__ ^ 

Example 9.2 (Nonexistence of a proper solution) Let p = 3 and m = 1, and suppose 
the random variables X U X 2 , and X } have the positive definite covariance matrix „ 

r i .9 . 7 i M 

X = .9 1 .4 

.7 .4 lj | 

Using the factor model in (9-4), we obtain 

X) - fii = ^ii^i + ei 

X 2 ” M2 ~ ^21^1 + e 2 

- fi 3 — Zi\Fi + e 3 

The covariance structure in (9-5) implies that 

X - LL' + ¥ 


1 — €11 + ipi 


.90 = € n € 21 .70 = € n € 31 

1 “ ^21 + 02 -40 = ^ 21^31 

1 = *31 + 0 3 


.70 — € n € 31 
.40 — € 21^31 


The pair of equations 


implies that 


Substituting this result for Z 2 i in the equation 

.90 = e u e 21 

yields P n = 1.575, or € n = ± 1.255. Since Var(Fx) = 1 (by assumption) and 
Var(X 1 ) = 1, € n = Cov^, FJ = Corr^.F,). Now, a correlation coefficient 
cannot be greater than unity (in absolute value), so, from this point of view, 
|* n | = 1.255 is too large. Also, the equation 

1 = ^11 +<Ai, or <A, = l-€?i 
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gives 

0! = 1 - L575 = -.575 

which is unsatisfactory, since it gives a negative value for Var (e x ) = 0j. 

Thus, for this example with m = 1, it is possible to get a unique numerical solu¬ 
tion to the equations X = LL' 4- ¥. However, the solution is not consistent with 
the statistical interpretation of the coefficients, so it is not a proper solution. ■ 


When m > 1, there is always some inherent ambiguity associated with the factor 
model. To see this, let T be any m X m orthogonal matrix^ so that TT' = T'T = I. 
Then the expression in (9-2) can be written 

X — ft = LF + e = LTT'F + e = L*F* + e (9-7) 

where 

L* = LT and F* = T'F 


Since 


E(F*) = T' E(F) = 0 

and 


Cov(F*) = T'Cov (F)T = T'T = I 

(mX;n) 


it is impossible, on the basis of observations on X, to distinguish the loadings L from 
the loadings L*. That is, the factors F and F* = T'F have the same statistical prop¬ 
erties, and even though the loadings L* are, in general, different from the loadings 
L, they both generate the same covariance matrix X. That is, 


X = LL' + ¥ = LTT'L' + ¥ = (L*) (L*)' + ¥ (9-8) 


This ambiguity provides the rationale for “factor rotation,” since orthogonal matrices 
correspond to rotations (and reflections) of the coordinate system for X. 


Factor loadings L are determined only up to an orthogonal matrix T. Thus, the 
loadings 

L* = LT and L (9-9) 

both give the same representation. The communalities, given by the diagonal 
elements of LL' = (L*) (L*)' are also unaffected by the choice of T. 


The analysis of the factor model proceeds by imposing conditions that allow 
one to uniquely estimate L and The loading matrix is then rotated (multiplied 
by an orthogonal matrix), where the rotation is determined by some “ease-of- 
interpretation” criterion. Once the loadings and specific variances are obtained, fac¬ 
tors are identified, and estimated values for the factors themselves (called factor 
scores ) are frequently constructed. 
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9.3 Methods of Estimation 

Given observations x 1 ,x 2 ,...,x„ onp generally correlated variables,factor analysis 
seeks to answer the question. Does the factor model of (9-4), with a small number of 
factors, adequately represent the data? In essence, we tackle this statistical model- 
building problem by trying to verify the covariance relationship in (9-5). 

The sample covariance matrix S is an estimator of the unknown population 
covariance matrix X. If the off-diagonal elements of S are small or those of the sample 
correlation matrix R essentially zero, the variables are not related, and a factor 
analysis will not prove useful. In these circumstances, the specific factors play the 
dominant role, whereas the major aim of factor analysis is to determine a few 
important common factors. 

If X appears to deviate significantly from a diagonal matrix, then a factor model 
can be entertained, and the initial problem is one of estimating the factor loadings 
and specific variances ^.We shall consider two of the most popular methods of para¬ 
meter estimation, the principal component (and the related principal factor) method 
and the maximum likelihood method. The solution from either method can be rotated 
in order to simplify the interpretation of factors, as described in Section 9.4. It is 
always prudent to try more than one method of solution; if the factor model is appro¬ 
priate for the problem at hand, the solutions should be consistent with one another. 

Current estimation and rotation methods require iterative calculations that must 
be done on a computer. Several computer programs are now available for this purpose. 

The Principal Component (and Principal Factor) Method 

The spectral decomposition of (2-16) provides us with one factoring of the covariance ma¬ 
trix X- Let X have eigenvalue-eigenvector pairs (A/, e,) with A t ^ A 2 ^ \ p ^ 0. 
Then 

X = A^ei + A 2 e 2 ei + • • • + k p e p e' p 

VX[ e\ 

= [VAje, J V^e 2 j ••• | VT p e p ] 


This fits the prescribed covariance structure for the factor analysis model having as 
many factors as variables (m = p) and specific variances <//, = 0 for all /.The load¬ 
ing matrix has ;th column given by VA ~j e y . That is, we can write 

X = L L' + 0 = LL' (9-11) 

(pXp) {p^p){p^p) (pXp) 

Apart from the scale factor VA^, the factor loadings on the ;th factor are the coeffi¬ 
cients for the ;th principal component of the population. 

Although the factor analysis representation of X in (9-11) is exact, it is not par¬ 
ticularly useful: It employs as many common factors as there are variables and does 
not allow for any variation in the specific factors e in (9-4). We prefer models that 
explain the covariance structure in terms of just a few common factors. One 
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approach, when the last p m eigenvalues are small, is to neglect the contribution 
°f ^m+i e m+i e m+i + " * + k p e p e' p to X in (9-10). Neglecting this contribution, we 
obtain the approximation 


X — [ \/Aj e 2 j x/A^ e 2 




VAiei 
VA^ e 2 


L L' (9-12) 

( pXm ) (m’Xp) 


The approximate representation in (9-12) assumes that the specific factors e in (9-4) 
are of minor importance and can also be ignored in the factoring of X. If specific 
factors are included in the model, their variances may be taken to be the diagonal 
elements of X - LL', where LL' is as defined in (9-12). 

Allowing for specific factors, we find that the approximation becomes 

X - LL' + * 


= [ | VX^e 2 j 


VAT ej 
VT 2 e 2 


*i 0 

0 fa 


where ^ ^ for i = 1,2,. ..,p. 

. 7=1 

To apply this approach to a data set Xj,x 2 ,..., x„, it is customary first to center 
the observations by subtracting the sample mean x. TTie centered observations 


x = X / 2 


X2 I X/2 “ X 2 


j - 1,2,...,/! 


have the same sample covariance matrix S as the original observations. 

In cases in which the units of the variables are not commensurate, it is usually 
desirable to work with the standardized variables 

( x n ~ *i) ~ 

V^n 

1 (* y2 ~ * 2 ) 

z j ~ ^22 j = 1 , 2 ,... ,n 


( x Jp x p) 


whose sample covariance matrix is the sample correlation matrix R of the observa¬ 
tions x 1 ,x 2 X a . Standardization avoids the problems of having one variable with 

large variance unduly influencing the determination of factor loadings. 
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The representation in (9-13), when applied to the sample covariance matrix S or 
the sample correlation matrix R, is known as the principal component solution. The 
name follows from the fact that the factor loadings are the scaled coefficients of the 
first few sample principal components. (See Chapter 8.) 


Principal Component Solution of the Factor Model 

The principal component factor analysis of the sample covariance matrix S is 

specified in terms of its eigenvalue-eigenvector pairs (Ai,^), (A 2 ,e 2 ). 

(A , e p ), where Aj > A 2 ^ ^ A p . Let m < p be the number of common fac¬ 

tors. Then the matrix of estimated factor loadings {€,•/} is given by 

L = [ VAj ej I VA^ej j j Vl^e m ] (9-15) 

The estimated specific variances are provided by the diagonal elements of the 


matrix S — LL', so 


o 

o 





$ = 

0 

</>2 • •' 0 

with 

<?, = Su 

m ~ 2 

(9-16) 






M 



_0 

0 • • • 4> P _ 





Communalities are estimated as 







hj = ii 1 + ti 2 + • • ■ 

■ + tfm 


(9-17) 


The principal component factor analysis of the sample correlation matrix is 
obtained by starting with R in place of S. 



For the principal component solution, the estimated loadings for a given 
factor do not change as the number of factors is increased. For example, if m = 1, 

L = [VXej], and if m = 2, L = [VA^e, j Va^], where (Aj.ej) and (A 2 ,ej) 
are the first two eigenvalue-eigenvector pairs for S (or R). 

By the definition^ the diagonal elements of S are equal to the diagonal 
elements of LL' + 'P.^However, the off-diagonal elements of S are not usually 
reproduced by LL' + ¥. How, then, do we select the number of factors ml 

If the number of common factors is not determined by a priori considerations, 
such as by theory or the work of other researchers, the choice of m can be based on 
the estimated eigenvalues in much the same manner as with principal components. 
Consider the residual matrix 

S - (LL' + $) (9-18) 

resulting from the approximation of S by the principal component solution. The diago¬ 
nal elements are zero, and if the other elements are also small, we may subjectively 
take the m factor model to be appropriate. Analytically, we have (see Exercise 9.5) 

Sum of squared entries of (S - (LL' + *^)) < A^,+i + • • * + X 2 P (9-19) 
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Consequently, a small value for the sum of the squares of the neglected eigenvalues 
implies a small value for the sum of the squared errors of approximation. 

Ideally, the contributions of the first few factors to the sample variances of the 
variables should be large. The contribution to the sample variance sa from the 
first common factor is The contribution to the total sample variance, s u + 
s 2 2 + ■ ■ * + s pp = tr(S), from the first common factor is then 


+ ?2i + ** 4 + t p \ - ( V^ei) (VAjei) = Aj 


since the eigenvector e! has unit length. In general, 


( Proportion of total 
sample variance 
due to yth factor 


/ a 

■*11 + *22 +_l- j for a factor analysis of S 

< . PP (9-20) 

Ay for a factor analysis of R 

P 


Criterion (9-20) is frequently used as a heuristic device for determining the appro¬ 
priate number of common factors. The number of common factors retained in the 
model is increased until a “suitable proportion” of the total sample variance has 
been explained. 

Another convention, frequently encountered in packaged computer programs, 
is to Set m equal to the number of eigenvalues of R greater than one if the sample 
correlation matrix is factored, or equal to the number of positive eigenvalues of S if 
the sample covariance matrix is factored. These rules of thumb should not be ap¬ 
plied indiscriminately. For example, m - p if the rule for S is obeyed, since all the 
eigenvalues are expected to be positive for large sample sizes. The best approach is 
to retain few rather than many factors, assuming that they provide a satisfactory in¬ 
terpretation of the data and yield a satisfactory fit to S or R. 


Example 9.3 (Factor analysis of consumer-preference data) In a consumer-preference 
study, a random sample of customers were asked to rate several attributes of a new 
product. The responses, on a 7-point semantic differential scale, were tabulated and 
the attribute correlation matrix constructed. The correlation matrix is presented next: 


Attribute (Variable) 


1 

2 

3 

4 

5 

Taste 

1 

‘1.00 

.02 

(96) 

.42 

.01" 

Good buy for money 

2 

.02 

1.00 

.13 

.71 


Flavor 

3 

.96 

.13 

1.00 

.50 

.11 

Suitable for snack 

4 

.42 

.71 

.50 

1.00 

® 

Provides lots of energy 

5 

.01 

.85 

.11 

.79 

1.00 _ 


It is clear from the circled entries in the correlation matrix that variables 1 and 
3 and variables 2 and 5 form groups. Variable 4 is “closer” to the (2,5) group than 
the (1,3) group. Given these results and the small number of variables, we might ex¬ 
pect that the apparent linear relationships between the variables can be explained in 
terms of, at most, two or three common factors. 
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The first two eigenvalues, \ x = 2.85 and A 2 = 1.81, of R are the only eigenval¬ 
ues greater than unity. Moreover, m = 2 common factors will account for a cumula¬ 
tive proportion 

\i + A 2 _ 2. 85 + 1.81 = 93 
P 5 

of the total (standardized) sample variance. The estimated factor loadings, commu- 
nalities, and specific variances, obtained using (9-15), (9-16), and (9-17), are given in 
Table 9.1. 


Table 9.1 


Variable 

Estimated factor 
loadings 

Fi | F 2 

Communalities 

h 2 i 

1. Taste 

.56 

.82 

.98 

2. Good buy 




for money 

.78 

-.53 

.88 

3. Flavor 

.65 

.75 

.98 

4. Suitable 




for snack 

.94 

-.10 

.89 

5. Provides 




lots of energy 

.80 

-.54 

.93 

Eigenvalues 

2.85 

1.81 


Cumulative 




proportion 




of total 




(standardized) 




sample variance 

.571 

.932 



Specific 

variances 

.02 


LL' + V = 


.56 

.82 

.78 

-.53 

.65 

.75 

.94 

-.10 

.80 

-.54 


".56 


.78 

.65 

.94 

.80* 

1 




.82 

— 

-.53 

.75 - 

-.10 

-.54 

J 




0 

0 

0 

0 


"1.00 

.01 

.97 

.44 

.00 

.12 

0 

0 

0 



1.00 

.11 

.79 

.91 

0 

.02 

0 

0 

= 



1.00 

.53 

.11 

0 

0 

.11 

0 





1.00 

.81 

0 

0 

0 

.07 J 






1.00 
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nearly reproduces the correlation matrix R. Thus, on a purely descriptive basis, we 
would judge a two-factor model with the factor loadings displayed in Table 9.1 as pro¬ 
viding a good fit to the data.The communalities (.98, .88, .98, .89, .93) indicate that the 
two factors account for a large percentage of the sample variance of each variable. 

We shall not interpret the factors at this point. As we noted in Section 9.2, the 
factors (and loadings) are unique up to an orthogonal rotation. A rotation of the 
factors often reveals a simple structure and aids interpretation. We shall consider 
this example again (see Example 9.9 and Panel 9.1) after factor rotation has been 
discussed. ■ 


Example 9.4 (Factor analysis of stock-price data) Stock-price data consisting of 
n — 103 weekly rates of return on p = 5 stocks were introduced in Example 8.5. 
In that example, the first two sample principal components were obtained from R. 
Taking m — 1 and m — 2, we can easily obtain principal component solutions to 
the orthogonal factor model. Specifically, the estimated factor loadings are the 
sample principal component coefficients (eigenvectors of R), scaled by the 
square root of the corresponding eigenvalues. The estimated factor loadings, 
communalities, specific variances, and proportion of total (standardized) sample 
variance explained by each factor for the m = 1 and m = 2 factor solutions are 
available in Table 9.2. The communalities are given by (9-17). So, for example, with 

m=2,h\ = + li 2 = (,732) 2 + (-.437) 2 = .73. 


Table 9.2 


One-factor solution Two-factor solution 


Variable 

Estimated factor 
loadings 

Fi 

Specific 

variances 

Estimated factor 
loadings 

Fi | F 2 

Specific 

variances 

ti = i - 

1. J P Morgan 

.732 

.46 

.732 

-.437 

.27 

2. Citibank 

.831 

.31 

.831 

-.280 

.23 

3. Wells Fargo 

.726 

.47 

.726 

-.374 

.33 

4. Royal Dutch Shell 

.605 

.63 

.605 

.694 

.15 

5. ExxonMobil 

.563 

.68 

.563 

.719 

.17 

Cumulative 
proportion of total 
(standardized) 
sample variance 
explained 

.487 


.487 

.769 



The residual matrix corresponding to the solution for m ~ 2 factors is 


0 

-.099 

-.185 

-.025 

.056 

-.099 

0 

-.134 

.014 

-.054 

-.185 

-.134 

0 

.003 

.006 

-.025 

.014 

.003 

0 

-.156 

.056 

-.054 

.006 

-.156 

0 
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The proportion of the total variance explained by the two-factor solution is appreciably 
larger than that for the one-factor solution. However, for m = 2, LL' produces 
numbers that are, in general, larger than the sample correlations. This is particularly 
true for r 13 . 

It seems fairly clear that the first factor, F { , represents general economic con¬ 
ditions and might be called a market factor. All of the stocks load highly on this fac¬ 
tor, and the loadings are about equal. The second factor contrasts the banking 
stocks with the oil stocks. (The banks have relatively large negative loadings, and ' 
the oils have large positive loadings, on the factor.) Thus, F 2 seems to differentiate 
stocks in different industries and might be called an industry factor. To summarize 
rates of return appear to be determined by general market conditions and activities 
that are unique to the different industries, as well as a residual or firm specific 
factor. This is essentially the conclusion reached by an examination of the sample 
principal components in Example 8.5. m 

A Modified Approach—the Principal Factor Solution 

A modification of the principal component approach is sometimes considered. We 
describe the reasoning in terms of a factor analysis of R, although the procedure is 
also appropriate for S. If the factor model p = LL' + ¥ is correctly specified, the 
m common factors should account for the off-diagonal elements of p, as well as 
the communality portions of the diagonal elements 

Pa = 1 = hj + <l>i 

If the specific factor contribution fa is removed from the diagonal or, equivalently, 
the 1 replaced by hf, the resulting matrix is p - V = LL'. 

Suppose, now, that initial estimates ip* of the specific variances are available. 
Then replacing thezth diagonal element of R by h* 2 = 1 - ip*, we obtain a “reduced” 
sample correlation matrix 


h? 

r \2 ' 

■' r >P 

r 12 

h? ■ 

r 2p 

n p 

r 2p '■ 

■ ■ /t* 2 

n P ■ 


Now, apart from sampling variation, all of the elements of the reduced sample cor 
relation matrix R r should be accounted for by the m common factors. In particular, 
R r is factored as 

R^LX (9-21) 

where L* = {£* } are the estimated loadings. 

The principal factor method of factor analysis employs the estimates 

L; = [V^eII VF m e* m ] 

(9-22) 

/=1 
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where (A,, e,), i 1,2,. .., m are the (largest) eigenvalue-eigenvector pairs deter- 
mined from R r . In turn, the communalities would then be (re)estimated by 

(9-23) 

;=i 

The principal factor solution can be obtained iteratively, with the communality esti¬ 
mates ot (9-23) becoming the initial estimates for the next stage. 

In the spirit of the principal component solution, consideration of the estimated 
eigenvalues Aj, Aj,..., A* helps determine the number of common factors to retain. 
An added complication is that now some of the eigenvalues may be negative, due to 
the use of initial communality estimates. Ideally, we should take the number of com¬ 
mon factors equal to the rank of the reduced population matrix. Unfortunately, this 
rank is not always well determined from R„ and some judgment is necessary. 

Although there are many choices for initial estimates of specific variances, the 
most popular choice, when one is working with a correlation matrix, is <//* = l/r“, 

where r“ is the ith diagonal element of R 1 The initial communality estimates then 
become J 

h- 2 = 1 - 0* = 1 - ~ (9.24) 

which is equal to the square of the multiple correlation coefficient between X L and 
* e °}* 2 r P 1 variables. The relation to the multiple correlation coefficient means 
that hj can be calculated even when R is not of full rank. For factoring S, the initial 
specific variance estimates use s“, the diagonal elements of S' 1 . Further discussion 
of these and other initial estimates is contained in [6]. 

Although the principal component method for R can be regarded as a principal 
factor method with initial communality estimates of unity, or specific variances 
equal to zero, the two are philosophically and geometrically different. (See [6].) In 
practice, however, the two frequently produce comparable factor loadings if the 
number of variables is large and the number of common factors is small. 

We do not pursue the principal factor solution, since, to our minds, the solution 
methods that have the most to recommend them are the principal component 
method and the maximum likelihood method, which we discuss next. 

The Maximum Likelihood Method 

If the common factors F and the specific factors e can be assumed to be normally 
distributed, then maximum likelihood estimates of the factor loadings and specific 
variancesmay be obtained. When Fy and e, are jointly normal, the observations 
A ; p - LF ) + Ej are then normal, and from (4-16), the likelihood is 

L(p,X) = (27r)“T|2| - §e - (|) tr [ 2 ~ I (£i (x-m)')] 

= (277) ~2 ~IX fo-i)(«r«>)] 

x ( 2 ^)-f|X| 4 e -(f)( i -'*)' 2 - 1 (-M) 


(9-25) 
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which depends on L and ¥ through 1 = LL' + This model is still not well 
defined, because of the multiplicity of choices for L made possible by orthogonal 
transformations. It is desirable to make L weU defined by imposing the computa- 
tionally convenient uniqueness condition 

L'^L = A a diagonal matrix (9-26) 

The maximum likelihood estimates L and ¥ must be obtained by numerical 
maximization of (9-25). Fortunately, efficient computer programs now exist that en- 

able one to get these estimates rather easily _ 

We summarize some facts about maximum likelihood estimators and, for now, 
rely on a computer to perform the numerical details. 

Result 9.1. Let X,,X 2> ...,X n be a random sample from N p (fi,X), where 
X = LL' + Hr is the covariance matrix for the m common factor model of (9-4). 
The maximum likelihood estimators L, ¥, and fi. = x maximize (9-25) subject to 
L’^-'L being diagonal. 

The maximum likelihood estimates of the communalities are 


hj — + €,2 + ’ • ‘ 

fort = 1,2,. 

..,p 

(9-27) 

Proportion of total sample \ 
variance due to;th factor j 


■ + Ki 

(9-28) 

*11 + *22 + •• 

' + s pp 


Proof. By the invariance property of maximum likelihood estimates (see Section 4.3), 
functions of L and Hr are estimated by the same functions of L and T'. In particu¬ 
lar, the communalities #«&+■■• + have maximum likelihood estimates 

hi — ii !+•••+ €irn- 

If, as in (8-10), the variables are standardized so that Z = V 1/2 (X - p). then 
the covariance matrix p of Z has the representation 



n = v _1 / 2 XV' 1/2 = (V 1/Z L) (V" 1/2 L)' + V'^VV 1/2 (9-29) 

* II 

Thus, p has a factorization analogous to (9-5) with loading matrix L* = V 1/2 L and f t 
specific variance matrix = \-^Hr\~^. By the invariance property of maxi- j : 
mum likelihood estimators, the maximum likelihood estimator of p is .i L 

p = (v _1/2 l)(v _1/2 l)' + v _1/2 'i'V' 1 ^ 2 --J [ 


= l,l; + Hr z (9 ' 30) 

where V' 1 ' 7 and L are the maximum likelihood estimators of \~ lp and L, respec¬ 
tively. (See Supplement 9A.) . ... ,. 

As a consequence of the factorization of (9-30), whenever the maximum likeli¬ 
hood analysis pertains to the correlation matrix, we call 

(9-31) 


h} = in + in + ■ 


• + iL 


i = 1 , 2 ,... 
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the maximum likelihood estimates of the communalities, and we evaluate the im¬ 
portance of the factors on the basis of 

f Proportion of total (standardized) 

^ sample variance due to jth factor 

To avoid more tedious notations, the preceding €,/s denote the elements of L z . 

Comment Ordinarily, the observations are standardized, and a sample corre¬ 
lation matrix is factor analyzed. The sample correlation matrix R is inserted for 
[(n - 1)/«]S in the likelihood function of (9-25), and the maximum likelihood 
estimates L z and ^ z are obtained using a computer. Although the likelihood in (9-25) is 
appropriate for S, not R, surprisingly, this practice is equivalent to obtaining the maxi¬ 
mum likelihood estimates L and 'P based on the sample covariance matrix S, setting 
L z = V~ 1/2 L and 4 r z = V _1/2 . HereV“ 1/2 is the diagonal matrix with the recip¬ 

rocal of the sample standard deviations (computed with the divisor Vn) on the main 
diagonal. 

Going in the other direction, given the estimated loadings L z and specific 
variances ^ z obtained from R, we find that the resulting maximum likelihood 
estimates for a factor analysis of the covariance matrix [(n - l)/n]S are 
L = V 1/2 L 2 and ¥ = V^^y 1 / 2 , or 


-2 -2 
h) + ^2j 


+ ... 


+ Ki 


(9-32) 


ill = and ipi = 4i ZJ a,i 

where <r ir is the sample variance computed with divisor n. The distinction between 
divisors can be ignored with principal component solutions. ■ 

The equivalency between factoring S and R has apparently been confused in 
many published discussions of factor analysis. (See Supplement 9A.) 


Example 9.5 (Factor analysis of stock-price data using the maximum likelihood 
method) The stock-price data of Examples 8.5 and 9.4 were reanalyzed assuming 
an m = 2 factor model and using the maximum likelihood method . The estimated 
factor loadings, communalities, specific variances, and proportion of total (stan¬ 
dardized) sample variance explained by each factor are in Table 9.3. 3 The corre¬ 
sponding figures for the m = 2 factor solution obtained by the principal component 
method (see Example 9.4) are also provided. The communalities corresponding to 
the maximum likelihood factoring of R are of the form [see (9-31)] h, = €,*j + € i2 - 
So, for example, 

h\ = (.115) 2 + (.765) 2 = .58 


3 The maximum likelihood solution leads to a Heywood case . For this example, the solution of the 
likelihood equations give estimated loadings such that a specific variance is negative.The software pro¬ 
gram obtains a feasible solution by slightly adjusting the loadings so that all specific variance estimates 
are nonnegative. A Heywood case is suggested here by the .00 value for the specific variance of Royal 
Dutch Shell. 
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Table 9.3 


Maximum likelihood 

Principal components j 


Estimated factor 

Specific 

Estimated factor 

Specific 


loadings 

variances 

loadings 

variances 

Variable 

Fi 

F 2 

A A 

</>, = !- hf 

Fi 

f 2 

ft" 

l 

i-H 

1! 

i-S 

1. J P Morgan 

.115 

.755 

.42 

.732 

-.437 

.27 

2. Citibank 

.322 

.788 

.27 

.831 

-.280 

.23 

3. Wells Fargo 

.182 

.652 

.54 

.726 

-.374 

.33 

4. Royal Dutch Shell 

1.000 

-.000 

.00 

.605 

.694 

.15 

5. Texaco 

.683 

-.032 

.53 

.563 

.719 

.17 

Cumulative 







proportion of total 







(standardized) 







sample variance 







explained 

.323 

.647 


.487 

.769 



The residual matrix is 


R - LL' - ¥ 


0 

.001 

-.002 

.000 

.052 

.001 

0 

.002 

.000 

-.033 

-.002 

.002 

0 

.000 

.001 

.000 

.000 

.000 

0 

.000 

.052 

-.033 

.001 

.000 

0 


The elements of R - LL' - 4 ? are much smaller than those of the residual matrix 
corresponding to the principal component factoring of R presented in Example 9.4. 
On this basis, we prefer the maximum likelihood approach and typically feature it in 
subsequent examples. 

The cumulative proportion of the total sample variance explained by the factors 
is larger for principal component factoring than for maximum likelihood factoring. 
It is not surprising that this criterion typically favors principal component factoring. 
Loadings obtained by a principal component factor analysis are related to the prin¬ 
cipal components, which have, by design, a variance optimizing property. [See the 
discussion preceding (8-19).] 

Focusing attention on the maximum likelihood solution, we see that all vari¬ 
ables have positive loadings on F x . We call this factor the market factor, as we did in 
the principal component solution. The interpretation of the second factor is not as 
clear as it appeared to be in the principal component solution. The bank stocks have 
large positive loadings and the oil stocks have negligible loadings on the second fac¬ 
tor F 2 . From this perspective, the second factor differentiaties the bank stocks from 
the oil stocks and might be called an industry factor. Alternatively, the second factor 
might be simply called a banking factor . 



The patterns of the initial factor loadings for the maximum likelihood solution 
are constrained by the uniqueness condition that be a diagonal matrix. 

Therefore, useful factor patterns are often not revealed until the factors are rotated 
(see Section 9.4). ■ 


Example 9.6 (Factor analysis of Olympic decathlon data) Linden [11] originally con¬ 
ducted a factor analytic study of Olympic decathlon results for all 160 complete 
starts from the end of World War II until the mid-seventies. Following his approach 
we examine the n = 280 complete starts from 1960 through 2004. The recorded 
values for each event were standardized and the signs of the timed events changed 
so that large scores are good for all events. We, too, analyze the correlation matrix, 
which based on all 280 cases, is 

R - 


1.000 

.6386 

.4752 

.3227 

.5520 

.3262 

.3509 

.4008 

.1821 

-.0352 

.6386 

1.0000 

.4953 

.5668 

.4706 

.3520 

.3998 

.5167 

.3102 

.1012 

.4752 

.4953 

1.0000 

.4357 

.2539 

.2812 

.7926 

.4728 

.4682 

-.0120 

.3227 

.5668 

.4357 

1.0000 

.3449 

.3503 

.3657 

.6040 

.2344 

.2380 

.5520 

.4706 

.2539 

.3449 

1.0000 

.1546 

.2100 

.4213 

.2116 

.4125 

.3262 

.3520 

.2812 

.3503 

.1546 

1.0000 

.2553 

.4163 

.1712 

.0002 

.3509 

.3998 

.7926 

.3657 

.2100 

.2553 

1.0000 

.4036 

.4179 

.0109 

.4008 

.5167 

.4728 

.6040 

.4213 

.4163 

.4036 

1.0000 

.3151 

.2395 

.1821 

.3102 

.4682 

.2344 

.2116 

.1712 

.4179 

.3151 

1.0000 

.0983 

-.0352 

.1012 

-.0120 

.2380 

.4125 

.0002 

.0109 

.2395 

.0983 

1.0000 


From a principal component factor analysis perspective, the first four eigen¬ 
values, 4.21,1.39,1.06, .92, of R suggest a factor solution with m = 3 or m — 4. A 
subsequent interpretation, much like Linden’s original analysis, reinforces the 
choice m = 4. 

In this case, the two solution methods produced very different results. For the prin¬ 
cipal component factorization, all events except the 1,500-meter run have large positive 
loading on the first factor. This factor might be labeled general athletic ability. Factor 2, 
which loads heavily on the 400-meter run and 1,500-meter run might be called a run - 
ning endurance factor. The remaining factors cannot be easily interpreted to our minds. 

For the maximum likelihood method, the first factor appears to be a general ath¬ 
letic ability factor but the loading pattern is not as strong as with principal compo¬ 
nent factor solution. The second factor is primarily a strength factor because shot put 
and discus load highly on this factor. The third factor is running endurance since the 
400-meter run and 1,500-meter run have large loadings. Again, the fourth factor is 
not easily identified, although it may have something to do with jumping ability or 
leg strength. We shall return to an interpretation of the factors in Example 9.11 after 
a discussion of factor rotation. 

The four-factor principal component solution accounts for much of the total 
(standardized) sample variance, although the estimated specific variances are 
large in some cases (for example, the javelin). This suggests that some events 
might require unique or specific attributes not required for the other events. The 
four-factor maximum likelihood solution accounts for less of the total sample 
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variance, but, as the following residual matrices indicate, the maximum likelihood 
estimates L and 'P do a better job of reproducing R than the principal component 
estimates L and 

Principal component: 

R - LL - ¥ = 


0 

-.082 

-.006 

-.021 

-.068 

.031 

-.016 

.003 

.039 

.062 

.082 

0 

-.046 

.033 

r- 

o 

i-H 

f 

-.078 

-.048 

-.059 

.042 

.006 

.006 

-.046 

0 

.006 

-.010 

-.014 

-.003 

-.013 

-.151 

.055 

.021 

.033 

.006 

0 

-.038 

-.204 

-.015 

-.078 

-.064 

-.086 

.068 

-.107 

-.010 

-.038 

0 

.096 

.025 

-.006 

.030 

-.074 

.031 

-.078 

-.014 

-.204 

.096 

0 

.015 

-.124 

.119 

.085 

.016 

-.048 

-.003 

-.015 

.025 

.015 

0 

-.029 

-.210 

.064 

.003 

• -.059 

-.013 

-.078 

-.006 

-.124 

-.029 

0 

-.026 

-.084 

.039 

.042 

-.151 

-.064 

.030 

.119 

-.210 

-.026 

0 

-.078 

.062 

.006 

.055 

-.086 

-.074 

.085 

.064 

-.084 

-.078 

0 


Maximum likelihood: 


R - LL' 

0 

- ¥ = 

.000 

.000 

.-.000 

-.000 

.000 

- .000 

.000 

-.001 

000 

.000 

0 

-.002 

.023 

.005 

.017 

-.003 

-.030 

.047 

-.024 

.000 

-.002 

0 

.004 

-.000 

-.009 

.000 

i 

8 
>—» 

-.001 

.000 

-.000 

.023 

.004 

0 

-.002 

-.030 

-.004 

-.006 

-.042 

.010 

-.000 

.005 

-.001 

-.002 

0 

-.002 

.001 

.001 

.000 

-.001 

.000 

l 

o 

)—11 

-.009 

-.030 

-.002 

0 

.022 

.069 

.029 

-.019 

-.000 

-.003 

.000 

-.004 

.001 

.022 

0 

-.000 

-.000 

.000 

.000 

-.030 

-.001 

-.006 

.001 

.069 

-.000 

0 

.021 

.011 

-.001 

.047 

-.001 

-.042 

.001 

.029 

-.000 

.021 

0 

-.003 

.000 

-.024 

.000 

.010 

-.001 

-.019 

.000 

.011 

-.003 

0 


A Large Sample Test for the Number of Common Factors 

Tlie assumption of a normal population leads directly to a test of the adequacy of 
the model. Suppose the m common factor model holds. In this case £ = LL' + 
and testing the adequacy of the m common factor model is equivalent to testing 

Hq: 1 = L V + V (9-33) 

(pXp) (pxm) (mXp) (pxp) 

versus any other positive definite matrix. When 2 does not have any special 
form, the maximum of the likelihood function [see (4-18) and Result 4.11 with 2 = 
((« — 1)/«)S = S„] is proportional to 

| S„ \~ n / 2 e~ np/2 


500 


(9-34) 
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Using Bartlett’s correction, we evaluate the test statistic in (9-39). 

|LL' + ¥| 

[n - 1 - (2p + 4m + 5)/6] In 

= |^103 — 1 — jin (1.0216) = 2.10 

Since i |(p -mf-p-m)= |[(5 - 2 ) 2 - 5 - 2] = 1. the 5% critical value 
2 / qc \ 2 _ 3 g 4 is not exceeded, and we fail to reject Ho- We conclude that the data do 
not contradict a two-factor model. In fact, the observed significance level or P-value, 
P[ x \ > 2.10] = .15 implies that H 0 would not be rejected at any reasonable level. * 

Large sample variances and covariances for the maximum likelihood estimates 
£.. 1. have been derived when these estimates have been determined from the sample 
covariance matrix S. (See [10].) The expressions are, in general, quite complicated. 

9.4 Factor Rotation 

As we indicated in Section 9.2, all factor loadings obtained from the initial loadings 
bv an orthogonal transformation have the same ability to reproduce the covanance 
(or correlation) matrix. [See (9-8).] From matrix algebra we know that an orthogo¬ 
nal transformation corresponds to a rigid rotation (or reflection of the coordinate 
axes For this reason, an orthogonal transformation of the factor loadings, as well as 
the implied orthogonal transformation of the factors, is called factor rotation. 

If L is the p X ni matrix of estimated factor loadings obtained by any method 
(principal component, maximum likelihood, and so forth) then 

L* = LT, where IT = T'T = I (9-42) 

is a p X m matrix of “rotated” loadings. Moreover, the estimated covariance (or 
correlation) matrix remains unchanged, since 

LL + V = LTT'L + * = L*L*’ + ¥ (9-43) 

Eauation (9-43) indicates that the residual matrix, S„ - LL' - ’F - S n - L*L* - ¥, 
remains unchanged. Moreover, the specific variances fc, and hence the communahti| 
h}, are unaltered. Thus, from a mathematical viewpoint, it is immaterial whether L 

01 Vn^th ^original loadings may not be readily interpretable, it is usual practice 
to rotate them until a “simpler structure” is achieved. The rationale is very much 
akin to sharpening the focus of a microscope in order to see the detail more clearly 
Ideally we should like to see a pattern of loadings such that each variable loads 
highly on a single factor and has small to moderate loadings on the remaining factors. 
However it is not always possible to get this simple structure, although the rotated load¬ 
ings for the decathlon data discussed in Example 9.11 provide a nearly ideal pattern. 

We shall concentrate on graphical and analytical methods for determining an 
orthogonal rotation to a simple structure. When m = 2, or the common factors are 
considered two at a time, the transformation to a simple structure can frequently 
determined graphically. The uncorrelated common factors are regarded as unit 
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vectors along perpendicular coordinate axes. A plot of the pairs of factor loadings 
(€ n , t i2 ) yields p points, each point corresponding to a variable.The coordinate axes 
can then be visually rotated through an angle—call it <j >—and the new rotated load¬ 
ings £*j are determined from the relationships 


where 



L* = 

(pX2) 

L T 

(^X2)(2X2) 


COS <j) 

sin <j> 

clockwise 

-sin <f> 

COS (j> 

rotation 

COS (f) 

—sin 4> 

counterclockwise 

sin <#> 

cos </> 

rotation 


(9-44) 


The relationship in (9-44) is rarely implemented in a two-dimensional graphical 
analysis. In this situation, clusters of variables are often apparent by eye, and these 
clusters enable one to identify the common factors without having to inspect the mag¬ 
nitudes of the rotated loadings. On the other hand, for m > 2, orientations are not 
easily visualized, and the magnitudes of the rotated loadings must be inspected to find 
a meaningful interpretation of the original data. The choice of an orthogonal matrix T 
that satisfies an analytical measure of simple structure will be considered shortly. 


Example 9.8 (A first look at factor rotation) Lawley and Maxwell [10] present the 
sample correlation matrix of examination scores in p ~ 6 subject areas for 
n = 220 male students. The correlation matrix is 


Gaelic English History Arithmetic Algebra Geometry 



.439 

1.0 


.410 

.351 

1.0 


.288 

.354 

.164 

1.0 


.329 

.320 

.190 

.595 

1.0 


.248 

.329 

.181 

.470 

.464 

1.0 


and a maximum likelihood solution for m = 2 common factors yields the estimates 
in Table 9.5. 
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All the variables have positive loadings on the first factor. Lawley and 
Maxwell suggest that this factor reflects the overall response of the students to in¬ 
struction and might be labeled a general intelligence factor. Half the loadings are 
positive and half are negative on the second factor. A factor with this pattern of 
loadings is called a bipolar factor. (The assignment of negative and positive poles 
is arbitrary, because the signs of the loadings on a factor can be reversed without 
affecting the analysis.) This factor is not easily identified, but is such that individu 
ais who get above-average scores on the verbal tests get above-average scores on 
the factor. Individuals with above-average scores on the mathematical tests get 
below-average scores on the factor. Perhaps this factor can be classified as 

“math-nonmath” factor. a ^ _ 

The factor loading pairs (€„, € i2 ) are plotted as points in Figure 9.1.The points 
are labeled with the numbers of the corresponding variables. Also shown is a clock 
wise orthogonal rotation of the coordinate axes through an angle of <f> = 20°. This 
angle was chosen so that one of the new axes passes through (i AU Z A1 ). When this is 
done, all the points fall in the first quadrant (the factor loadings are all positive), and 
the two distinct clusters of variables are more clearly revealed. 

The mathematical test variables load highly on F\ and have negligible load 
ings on F* 2 . The first factor might be called a mathematical-ability factor. Similarly, 
the three verbal test variables have high loadings on F 2 and moderate to small 
loadings on F\. The second factor might be labeled a verbal-ability factor. 
The general-intelligence factor identified initially is submerged in the factors F v 
and F* 

The rotated factor loadings obtained from (9-44) with <j> = 20° and the 
corresponding communality estimates are shown in Table 9.6. The magnitudes of 
the rotated factor loadings reinforce the interpretation of the factors suggested by 

Figure 9.1. , . ^ . 

The communality estimates are unchanged by the orthogonal rotation, since 
LL' = LTT'L' = L*L*', and the communalities are the diagonal elements of these 

matrices. ,. . 

We point out that Figure 9.1 suggests an oblique rotation of the coordinates. 

One new axis would pass through the cluster {1,2,3} and the other through the 
{4 5 6} group. Oblique rotations are so named because they correspond to a 
nonrigid rotation of coordinate axes leading to new axes that are not perpendicular. 




0 ~s r.o F] 


Figure 9.1 Factor rotation for test 
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It is apparent, however, that the interpretation of the oblique factors for this 
example would be much the same as that given previously for an orthogonal 
rotation. m 


Kaiser [9] has suggested an analytical measure of simple structure known as the 
varimax (or normal varimax) criterion. Define £*j = €* ; /A x to be the rotated coeffi¬ 
cients scaled by the square root of the communalities. Then the (normal) varimax 
procedure selects the orthogonal transformation T that makes 


i m r p ~ * 

- P % L§< 


as large as possible. 

Scaling the rotated coefficients £*j has the effect of giving variables with small 
communalities relatively more weight in the determination of simple structure. 
After the transformation T is determined, the loadings Z*j are multiplied by hi so 
that the original communalities are preserved. 

Although (9-45) looks rather forbidding, it has a simple interpretation. In 
words, 


y ^ ^ / variance of squares of (scaled) loadings for 
ftl \ jth factor 

Effectively, maximizing V corresponds to “spreading out” the squares of the load¬ 
ings on each factor as much as possible. Therefore, we hope to find groups of large 
and negligible coefficients in any column of the rotated loadings matrix L*. 

Computing algorithms exist for maximizing V, and most popular factor analysis 
computer programs (for example, the statistical software packages SAS, SPSS, 
BMDP, and MINITAB) provide varimax rotations. As might be expected, varimax 
rotations of factor loadings obtained by different solution methods (principal com¬ 
ponents, maximum likelihood, and so forth) will not, in general, coincide. Also, the 
pattern of rotated loadings may change considerably if additional common factors 
are included in the rotation. If a dominant single factor exists, it will generally be ob¬ 
scured by any orthogonal rotation. By contrast, it can always be held fixed and the 
remaining factors rotated. 


(9-46) 



508 Chapter 9 Factor Analysis and Inference for Structured Covariance Matrices 


Example 9.9 (Rotated loadings for the consumer-preference data) Let us return to 
the marketing data discussed in Example 9.3.The original factor loadings (obtained 
by the principal component method), the communalities, and the (varimax) rotated 
factor loadings are shown in Table 9.7. (See the SAS statistical software output in 
Panel 9.1.) 


Table 9.7 



Estimated j 

Rotated 


factor 


estimated factor 


loadings 


loadings 

Variable 

Fx 

Ft 

F\ 

Ft 

1. Taste 

.56 

.82 

1 .02 


2. Good buy for money 

.78 

-.52 


-.01 

3. Flavor 

.65 

.75 

.13 

(§) 

4. Suitable for snack 

.94 

.10 

(M\ 

.43 

5. Provides lots of energy 

.80 

-.54 

{&) 

-.02 

Cumulative proportion 
of total (standardized) 


i 


.932 

sample variance explained 

.571 

.932 

.507 


Communalities 

h} 


It is clear that variables 2, 4, and 5 define factor 1 (high loadings on factor 1, 
small or negligible loadings on factor 2), while variables 1 and 3 define factor 2 (high 
loadings on factor 2, small or negligible loadings on factor 1). Variable 4 is most 
closely aligned with factor 1, although it has aspects of the trait represented by 
factor 2. We might call factor 1 a nutritional factor and factor 2 a taste factor. 

The factor loadings for the variables are pictured with respect to the original 
and (varimax) rotated factor axes in Figure 9.2. ■ 


/ 

A l 

/ • 3 


5 " / 

/ 

/ 

/ 

o 4- 


^ Figure 9.2 Factor rotation for 
F*i hypothetical marketing data. 
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PANEL 9.1 SAS ANALYSIS FOR EXAMPLE 9.9 USING PROC FACTOR. 


title 'Factor Analysis'; 
data consumer(type = corr); 

_type_='CORR'; 

input _name_$ taste money flavor snack energy; 
cards; 

taste 1.00 

money .02 1.00 

flavor .96 .13 1.00 

snack .42 .71 .50 1.00 

energy .01 .85 .11 .79 1.00 


PROGRAM COMMANDS 


proc factor res data=consumer 

method=prin nfact=2rotate=varimax preplot plot; 
var taste money flavor snack energy; 


Initial Factor Method: Principal Components 


Eigenvalue 

Difference 


Prior Communality Estimates: ONE 
Eigenvalues of the Correlation Matrix: Total = 5 Average = 1 


2.853090 

1.046758 


2 

1.806332 

1.601842 


0.204490 

0.102081 


4 

0.102409 

0.068732 


Proportion 

0.5706 

0.3613 

Cumulative 

0.5706 

0.9319 


2 factors will be retained by the NFACTOR criterion. 


5 

0.033677 


Factor Pattern 


TASTE 

MONEY 

FLAVOR 

SNACK 


FACTOR 1 

FACTOR2 

0.55986 

0.81610 

0.77726 

-0,52420 

0.64534 

0.74795 

0.93911 

-0.10492 

0.79821 

-0.54323 


Final Communality Estimates: Total = 4.659423 

TASTE MONEY FLAVOR SNACK ENERGY 

0.97961 0.878920 0.975883 0.892928 ^ 6,932231 I 


(continues on next page) 
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panel 9.1 ( continued) 


Rotation Method: Varimax 


Rotated Factor Pattern 



FACTOR1 

FACTOR2 

TASTE 

0.01970 

0.98948 

MONEY 

0.93744 

-0.01123 

FLAVOR ' 

0.12856 

0.97947 

SNACK 

0.84244 

0.42805 

ENERGY 

0.96539 

-0.01563 


Variance explained by each factor 

FACTOR 1 FACTOR2 

2.537396 2.122027 


Rotation of factor loadings is recommended particularly for loadings 
obtained by maximum likelihood,since the initial values are constrained to satisfy 
the uniqueness condition that L'* _, L be a diagonal matrix. This condition is 
convenient for computational purposes, but may not lead to factors that can easily 
be interpreted. 


Example 9.10 (Rotated loadings for the stock-price data) Table 9.8 shows the initial 
and rotated maximum likelihood estimates of the factor loadings for the stock-price 
data of Examples 8.5 and 9.5. An m = 2 factor model is assumed. The estimated 


Table 9.8 


- Variable 

J P Morgan 
Citibank 
Wells Fargo 
Royal Dutch Shell 
ExxonMobil 

Cumulative 
proportion 
of total 

sample variance 
explained 


Maximum likelihood 
estimates of factor 
loadings 


Rotated estimated 
factor loadings 



Specific 

variances 

jf} - 1 - hf 


.346 .647 
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specific variances and cumulative proportions of the total (standardized) sample vari¬ 
ance explained by each factor are also given. 

An interpretation of the factors suggested by the unrotated loadings was pre¬ 
sented in Example 9.5. We identified market and industry factors. 

The rotated loadings indicate that the bank stocks (JP Morgan, Citibank, and 
Wells Fargo) load highly on the first factor, while the oil stocks (Royal Dutch 
Shell and ExxonMobil) load highly on the second factor. (Although the rotated 
loadings obtained from the principal component solution are not displayed, the 
same phenomenon is observed for them.) The two rotated factors, together, 
differentiate the industries. It is difficult for us to label these factors intelligently. 
Factor 1 represents those unique economic forces that cause bank stocks to 
move together. Factor 2 appears to represent economic conditions affecting oil 
stocks. 

As we have noted, a general factor (that is, one on which all the variables load 
highly) tends to be “destroyed after rotation.” For this reason, in cases where a gen¬ 
eral factor is evident, an orthogonal rotation is sometimes performed with the gen¬ 
eral factor loadings fixed . 5 ■ 


Example 9.11 (Rotated loadings for the Olympic decathlon data) The estimated 
factor loadings and specific variances for the Olympic decathlon data were 
presented in Example 9.6. These quantities were derived for an m = 4 factor 
model, using both principal component and maximum likelihood solution 
methods. The interpretation of all the underlying factors was not immediately 
evident. A varimax rotation [see (9-45)] was performed to see whether the rotated 
factor loadings would provide additional insights. The varimax rotated loadings 
for the m = 4 factor solutions are displayed in Table 9.9, along with the specific 
variances. Apart from the estimated loadings, rotation will affect only the distribu¬ 
tion of the proportions of the total sample variance explained by each factor. The 
cumulative proportion of the total sample variance explained for all factors does 
not change. 

The rotated factor loadings for both methods of solution point to the same 
underlying attributes, although factors 1 and 2 are not in the same order. We see 
that shot put, discus, and javelin load highly on a factor, and, following Linden 
[11], this factor might be called explosive arm strength. Similarly, high jump, 
110 -meter hurdles, pole vault, and—to some extent—long jump load highly on 
another factor. Linden labeled this factor explosive leg strength. The 100 -meter 
run, 400-meter run, and—again to some extent—the long jump load highly on a 
third factor. This factor could be called running speed. Finally, the 1500-meter run 
loads heavily and the 400-meter run loads heavily on the fourth factor. Linden 
called this factor running endurance. As he notes, “The basic functions indicated in 
this study are mainly consistent with the traditional classification of track and 
field athletics.” 


5 Some general-purpose factor analysis programs allow one to fix loadings associated with certain 
factors and to rotate the remaining factors. 
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Table 9.9 __ 


Variable 


Principal component 
Estimated 

rotated ^ Sj 

factor loadings, €*• vai 

F\ F\ F* 3 F\ 


Specific 

variances 


Maximum likelihood 
Estimated 

rotated A Specific 

factor loadings, variances 



.267 .221 176831 .293 

.086 [7471 .068 [507 


254 [739] .057 .242 

.142 .151 [519] [700] 


110 -m 

hurdles 

Discus 


Javelin 

1500-m 

run 

Cumulative 

proportion 

of total 

sample 

variance 

explained 


.048 .108 [78261 —161 
[8321 .185 .204 -.016 


.136 1.4651 .173 -.033 
[7931 .220 .133 -.009 


324 ,278 [656] .293 

7541 .024 .054 .188 


314 [613] .169 .279 
477] .160 .041 .139 


.002 .019 .075 


.001 .110 -.070 p 


.22 .43 .62 .76 


.20 .37 .51 .62 


Plots of rotated maximum likelihood loadings for factors pairs (1.2) 
and (1,3) are displayed in Figure 9.3 on page 513. The points are generally 
grouped along the factor axes. Plots of rotated principal component loadings are 
very similar. m 


Oblique Rotations 

Orthogonal rotations are appropriate for a factor mode) in which the common fee 
tors are assumed to be independent. Many investigators in social sciences consider 
oblique (nonorthogonal) rotations, as well as orthogonal rotations. The former are 



Factor 2 
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0,0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.6 0.8 


Factor 1 Factor 1 

Figure 9.3 Rotated maximum likelihood loadings for factor pairs (1,2) and (1, 3)— 
decathlon data. (The numbers in the figures correspond to variables.) 

often suggested after one views the estimated factor loadings and do not follow 
from our postulated model. Nevertheless, an oblique rotation is frequently a useful 
aid in factor analysis. 

If we regard the m common factors as coordinate axes, the point with the m 
coordinates (€ n , € j2 , .... €,* m ) represents the position of the tth variable in the factor 
space . Assuming that the variables are grouped into nonoverlapping clusters, an or¬ 
thogonal rotation to a simple structure corresponds to a rigid rotation of the coordi : 
nate axes such that the axes, after rotation, pass as closely to the clusters as possible. 
An oblique rotation to a simple structure corresponds to a nonrigid rotation of the 
coordinate system such that the rotated axes (no longer perpendicular) pass (near¬ 
ly) through the clusters. An oblique rotation seeks to express each variable in terms 
of a minimum number of factors—preferably, a single factor. Oblique rotations are 
discussed in several sources (see, for example, [6] or [10]) and will not be pursued in 
this book. 


9.5 Factor Scores 

In factor analysis, interest is usually centered on the parameters in the factor model. 
However, the estimated values of the common factors, called factor scores , may also 
be required. These quantities are often used for diagnostic purposes, as well as in¬ 
puts to a subsequent analysis. 

Factor scores are not estimates of unknown parameters in the usual sense. 
Rather, they are estimates of values for the unobserved random factor vectors Fy, 
j = 1,2,..., n. That is, factor scores 

fy = estimate of the values fy attained by Fy (y'th case) 
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The estimation situation is complicated by the fact that the unobserved quantities f 
and £j outnumber the observed x,-. To overcome this difficulty, some rather heuris¬ 
tic, but reasoned, approaches to the problem of estimating factor values have been 
advanced. We describe two of these approaches. 

Both of the factor score approaches have two elements in common: 

1. They treat the estimated factor loadings € /; - and specific variances fa as if they 
were the true values. 

2. They involve linear transformations of the original data, perhaps centered 
or standardized. Typically, the estimated rotated loadings, rather than the 
original estimated loadings, are used to compute factor scores. The com¬ 
putational formulas, as given in this section, do not change when rotated load¬ 
ings are substituted for unrotated loadings, so we will not differentiate 
between them. 


The Weighted Least Squares Method 

Suppose first that the mean vector /t, the factor loadings L, and the specific variance 
¥ are known for the factor model 

X - ft = L F+e 

(pXl) (pXl) (pXm)(mXl) (pXl) 

Further, regard the specific factors s’ - [ex, e 2 ,...,e p ] as errors. Since 
Var(e ( ) = ip h i = 1,2,...,/?, need not be equal, Bartlett [2] has suggested that 
weighted least squares be used to estimate the common factor values. 

The sum of the squares of the errors, weighted by the reciprocal of their 
variances, is 

p 

2f = £'¥ _1 E = (* - ft - Lfy'TH* - n - Lf) (9-47) 

1=1 Yi 

Bartlett proposed choosing the estimates f of f to minimize (9-47). The solution (see 
Exercise 7.3) is 

f = (L'¥- I L)' 1 L'*- 1 (x - m) (9-48) 

Motivated by (9^48), we take the estimates L, and jx = x as the true values and 
obtain the factor scores for the yth case as 

tj = (L'$- 1 L)~ 1 L'¥- 1 (x / - x) (9-49) 

When L and ¥ are determined by the maximum likelihood method, these estimates 
must satisfy the uniqueness condition, L'^^L = A, a diagonal matrix. We then 
have the following: 
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Factor Scores Obtained by Weighted Least Squares 
from the Maximum Likelihood Estimates 

tj = (L'^Lr 1 !.'^- 1 ^. - fi) 

= i - x), j = 1,2,...,n 

or, if the correlation matrix is factored 

tj = (L;*- 1 !,,)- 1 !;*- 1 *, 

= A^Li^-’zy, / = 1,2,..., ft 

where z t = D" 1 ^ - x), as in (8-25), and p = L z L'i + V t . 


The factor scores generated by (9-50) have sample mean vector 0 and zero sample 
covariances. (See Exercise 9.16.) 

If rotated loadings L* = LT are used in place of the original loadings in (9-50), 
the subsequent factor scores, f*, are related to f, by f* = T # f ; , / = 1,2,..., ai. 

Comment. If the factor loadings are estimated by the principal component 
method, it is customary to generate factor scores using an unweighted (ordinary) 
least squares procedure. Implicitly, this amounts to assuming that the ^ are equal or 
nearly equal. The factor scores are then 

== (L'L)" 1 L'(x > - x) 
or 

f y = (l;l b )- 1 l;z > 

for standardized data. Since L = [V^e, i V^e 2 ; ; V^e m ] [see (9-15)], 

we have J 
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Comparing (9-51) with (8-21), we see that the fy are nothing more than the first m 
(scaled) principal components, evaluated at x y . 

The Regression Method 

Starting again with the original factor model X - p = LF + e, we initially treat 
the loadings matrix L and specific variance matrix ¥ as known. When the common 
factors F and the specific factors (or errors) e are jointly normally distributed with 
means and covariances given by (9-3), the linear combination X - p = LF + e has 
an N p (0, LL' + ¥) distribution. (See Result 4.3.) Moreover, the joint distribution 
of (X - fx) and F is N m+P ( 0, X*), where 


(m+p)X(m+p) 


X = LL' + ¥ j 

L 

(P*P) \ 

(pxm) 

L' 

I 

(mXp) 

(rnXrn) 


and 0 is an (m + p) X 1 vector of zeros. Using Result 4.6, we find that the condi¬ 
tional distribution of F|x is multivariate normal with 

mean = £(F|x) = L'X _1 ( X - ft) ~ L'(LL' + ¥) 7 (x - p) (9-53) 


covariance = Cov(F|x) = I — L'X *L = I _ L'(LL' + T') L (9-54) 

The quantities L'(LL' + 'R )~ 1 in (9-53) are the coefficients in a (multivariate) re¬ 
gression of the factors on the variables. Estimates of these coefficients produce 
factor scores that are analogous to the estimates of the conditional mean values in 
multivariate regression analysis. (See Chapter 7.) Consequently, given any vector of 
observations x,, and taking the maximum likelihood estimates L and ¥ as the true val¬ 
ues, we see that the ;th factor score vector is given by 

tj = i/irVxy - x) = £/(££/ + ^) _1 (xy-x), j = 1,2,...,^ (9-55) 

The calculation of f y in (9-55) can be simplified by using the matrix identity (see 
Exercise 9.6) 

L' (LL' + ’i'T 1 = (I + L’i '" 1 !)' 1 L' ¥ _1 (9-56) 

(mXp) ( pXp ) (mXw) (mXp)(pXp) 

This identity allows us to compare the factor scores in (9-55), generated by the re¬ 
gression argument, with those generated by the weighted least squares procedure 

[see (9-50)]. Temporarily, we denote the former by if and the latter by Then, 
using (9-56), we obtain 

tf s = (L'^ _l L) _1 (I + L'^ -1 !)^ = (I + (L'¥ - 1 L) _ 1 )f? (9-57) 

For maximum likelihood estimates (L' 1 i ,- 1 L )' 1 = A 1 and if the elements of this 
diagonal matrix are close to zero, the regression and generalized least squares 
methods will give nearly the same factor scores. 
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In an attempt to reduce the effects of a (possibly) incorrect determination of 
the number of factors, practitioners tend to calculate the factor scores in (9-55) by 
using S (the original sample covariance matrix) instead of X = LL' + “'F. We then 
have the following: 


Factor Scores Obtained by Regression 

f; = L'S _1 (x ; - x), j = 1,2,.... n 

or, if a correlation matrix is factored, (9-58) 

f) = Li R^Zy, j = 1,2,..., n 

where, see (8-25), 

z y = D“ 1/2 (xy — x) and p ~ L Z V Z + 

Again, if rotated loadings L* = LT are used in place of the original loadings in 
(9-58), the subsequent factor scores fy are related to fy by 

fy = T'fy, j = l,2,...,n 

A numerical measure of agreement between the factor scores generated from 
two different calculation methods is provided by the sample correlation coefficient 
between scores on the same factor. Of the methods presented, none is recommended 
as uniformly superior. 

Example 9.12 (Computing factor scores) We shall illustrate the computation of fac¬ 
tor scores by the least squares and regression methods using the stock-price data 
discussed in Example 9.10. A maximum likelihood solution from R gave the esti¬ 
mated rotated loadings and specific variances 

“.763 .024“ 

.821 .227 

L* = .669 .104 and 

.118 .993 
.113 .675 _ 

The vector of standardized observations, 

z' = [.50, -1.40, -.20, -.70,1.40] 
yields the following scores on factors 1 and 2 : 


”.42 0 0 0 0 “ 

0 .27 0 0 0 

= 0 0 .54 0 0 

0 0 0 .00 0 

0 0 0 0 .53 _ 
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Weighted least squares (9-50) : 6 

f = 

Regression (9-58): 


’= LlR'z = 


331 .526 .221 -.137 .011 
040 -.063 -.026 1.023 -.001 



In this case, the two methods produce very similar results. All of the regrJgg 
factor scores, obtained using (9-58), are plotted in Figure 9.4. 


Comment . Factor scores with a rather pleasing intuitive property can be tori 
structed very simply. Group the variables with high (say, greater than .40j 
absolute value) loadings on a factor. The scores for factor 1 are then formedT 
summing the (standardized) observed values of the variables in the group, con 
bined according to the sign of the loadings. The factor scores for factor 2 ar|§T 
sums of the standardized observations corresponding to variables with high loadin 
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Figure 9.4 Factor scores using (9-58) for factors 1 and 2 of the stock-p 
(maximum likelihood estimates of the factor loadings). 

6 In order to calculate the weighted least squares factor scores, .00 in the fourth diagonal| 
was set to .01 s 0 that this matrix could be inverted. 
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on factor 2, and so forth. Data reduction is accomplished by replacing the stan¬ 
dardized data by these simple factor scores. The simple factor scores are frequently 
highly correlated with the factor scores obtained by the more complex least 
squares and regression methods. 


Example 9.13 (Creating simple summary scores from factor analysis groupings) The 

principal component factor analysis of the stock price data in Example 9.4 produced 
the estimated loadings 


".732 

-.437' 


".852 

.030 

.831 

-.280 


.851 

.214 

.726 

-.374 

and L* = LT = 

.813 

.079 

.605 

.694 


.133 

.911 

.563 

.719 _ 


_.084 

.909 _ 


For each factor, take the loadings with largest absolute value in L as equal in magni¬ 
tude, and neglect the smaller loadings. Thus, we create the linear combinations 

fl = x l + x 2 + x 3 + x 4 + *5 
f 2 = X 4 + X 5 ~ 

as a summary. In practice, we would standardize these new variables. 

It instead of L, we start with the varimax rotated loadings L*, the simple factor 
scores would be 

fl = Xl + x 2 + x 3 
f 2 = x 4 + x 5 

The identification of high loadings and negligible loadings is really quite subjective. 
Linear compounds that make subject-matter sense are preferable. ■ 

Although multivariate normality is often assumed for the variables in a factor 
analysis, it is very difficult to justify the assumption for a large number of variables. 
As we pointed out in Chapter 4, marginal transformations may help. Similarly, the 
factor scores may or may not be normally distributed. Bivariate scatter plots of fac¬ 
tor scores can produce all sorts of nonelliptical shapes. Plots of factor scores should 
be examined prior to using these scores in other analyses. They can reveal outlying 
values and the extent of the (possible) nonnormality. 


Perspectives and a Strategy for Factor Analysis 

There are many decisions that must be made in any factor analytic study. Probably 
the most important decision is the choice of m, the number of common factors. 
Although a large sample test of the adequacy of a model is available for a given m y it 
is suitable only for data that are approximately normally distributed. Moreover, the 
test will most assuredly reject the model for small m if the number of variables and 
observations is large. Yet this is the situation when factor analysis provides a useful 
approximation. Most often, the final choice of m is based on some combination of 
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(1) the proportion of the sample variance explained, (2) subject-matter knowledge, 

and ( 3 ) the “reasonableness” of the results. 

The choice of the solution method and type of rotation is a less crucial deci¬ 
sion In fact, the most satisfactory factor analyses are those in which rotations are 
tried with more than one method and all the results substantially confirm the same 

factor structure. f . , 

At the present time, factor analysis still maintains the flavor of an art, and no 

single strategy should yet be “chiseled into stone.” We suggest and illustrate one 

reasonable option: 

1 Perform a principal component factor analysis. This method is particularly 

* appropriate for a first pass through the data. (It is not required that R or S be 
nonsingular.) 

(a) Look for suspicious observations by plotting the factor scores. Also, 
calculate standardized scores for each observation and squared distances as 
described in Section 4.6. 

(b) Try a varimax rotation. 

2. Perform a maximum likelihood factor analysis, including a varimax rotation. 

3. Compare the solutions obtained from the two factor analyses. 

(a) Do the loadings group in the same manner? 

(b) Plot factor scores obtained for principal components against scores from 
the maximum likelihood analysis. 

4 Repeat the first three steps for other numbers of common factors m. Do extra fac- 

* tors necessarily contribute to the understanding and interpretation of the data? 

5. For large data sets, split them in half and perform a factor analysis on each part. 
Compare the two results with each other and with that obtained from the com¬ 
plete data set to check the stability of the solution. (The data might be divided 
by placing the first half of the cases in one group and the second half of the 
cases in the other group. This would reveal changes over time.) 


Example 9.14 (Factor analysis of chicken-bone data) We present the results of sev¬ 
eral factor analyses on bone and skull measurements of white leghorn fowl. The 
original data were taken from Dunn [5]. Factor analysis of Dunn’s data was orig¬ 
inally considered by Wright [15], who started his analysis from a different corre¬ 
lation matrix than the one we use. 

The full data set consists of n = 276 measurements on bone dimensions: 


Head: 

Leg: 

Wing: 


f X\ ~ skull length 
[X 2 = skull breadth 

j X 3 = femur length 
\x 4 = tibia length 

jx 5 = humerus length 
[X 6 = ulna length 
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The sample correlation matrix 


1.000 

.505 

.569 

.602 

.621 

.603 

.505 

1.000 

.422 

.467 

.482 

.450 

.569 

.422 

1.000 

.926 

.877 - 

.878 

.602 

.467 

.926 

1.000 

.874 

.894 

.621 

.482 

.877 

.874 

1.000 

.937 

.603 

.450 

.878 

.894 

.937 

1.000 


was factor analyzed by the principal component and maximum likelihood methods 
for an m = 3 factor model. The results are given in Table 9.10 . 7 


Table 9.10 Factor Analysis of Chicken-Bone Data 


Principal Component 



Estimated factor loadings 

Rotated estimated loadings 

I 

Variable 

F, 

F 2 

Fi 


F[ 

Fl 

Fl 


1. Skull length 

.741 

.350 

.573 


.355 

.244 

(.902) 

.00 

2. Skull breadth 

.604 

.720 

-.340 


.235 

(3 49) 

.211 

.00 

3. Femur length 

.929 

-.233 

-.075 


'.9211 

.164 

.218 

.08 

4. Tibia length 

.943 

-.175 

-.067 


.904 

.212 

.252 

.08 

5. Humerus length 

.948 

-.143 

-.045 


.888 

.228 

.283 

.08 

6 . Ulna length 

.945 

-.189 

-.047 


1-908J 

.192 

.264 

.07 

Cumulative 









proportion of 
totd (standardized) 
sample variance 
explained 

.743 

.873 

.950 


.576 

.763 

.950 


Maximum Likelihood 


Variable 

1. Skull length 

2. Skull breadth 

3. Femur length 

4. Tibia length 

5. Humerus length 

6 . Ulna length 

Cumulative 
proportion of 
total (standardized) 
sample variance 
explained 


Estimated factor loadings Rotated estimated loadings 
Fi F 2 F 3 F[ F\ F' 3 


.738 .823 


.559 .779 


7 Notice the estimated specific variance of .00 for tibia length in the maximum likelihood solution. 
This suggests that maximizing the likelihood function may produce a Hey wood case. Readers attempting 
to replicate our results should try the Hey(wood) option if SAS or similar software is used. 
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After rotation, the two methods of solution appear to give somewhat different 
results Focusingour attention on the principal component method and the cumula¬ 
tive proportion of the total sample variance explained, we see that a three-factor so¬ 
lution appears to be warranted. The third factor explains a “significant” amount of 
additional sample variation. The first factor appears to be a body-size factor domi¬ 
nated bv wing and leg dimensions. The second and third factors, collectively, repre¬ 
sent skull dimensions and might be given the same names as the variables, skull 
breadth and skull length, respectively. 

The rotated maximum likelihood factor loadings are consistent with those gen- 
erated bv the principal component method for the first factor, but not for factors 2 
and 3 For the maximum likelihood method, the second factor appears to represent 
head size The meaning of the third factor is unclear, and it is probably not needed. 

Further support for retaining three or fewer factors is provided by the residual 
matrix obtained from the maximum likelihood estimates: 



.000 






-.000 

.000 





-.003 

.001 

.000 



*« = 

.000 

.000 

.000 

.000 



-.001 

.000 

.000 

.000 

.000 


.004 

-.001 

-.001 

.000 

-.000 


r - LX - Vz 


All of the entries in this matrix are very small. We shall pursue the m- 3 factor 
mcLe in dris example. An * = 2 factor model is considered in Exercise 9.10. 

Factor scores for factors 1 and 2 produced from (9-58) with the rotated maxi¬ 
mum likelihood estimates are plotted in Figure 9.5. Plots of this kind allow us to 
identify observations that, for one reason or another, are not consistent with the 

remaining observations. Potential outliers are circled in the figure. 

It is also of interest to plot pairs of factor scores obtained using the principal 
component and maximum likelihood estimates of factor loadings. For thejicken 
bone data plots of pairs of factor scores are given in Figure 9.6 on pages 524-526 If 
the loadings on a particular factor agree, the pairs of scores should cluster tightly 
about the 45° line through the origin. Sets of loadings that do not agree will produce 
fecmr scores that deviate from this pattern. If the latter occurs, it is usually associat¬ 
ed with the last factors and may suggest that the number of factors is too [arg^That 
is the last factors are not meaningful. This seems to be the case with the third factor 
in the chicken-bone data, as indicated by Plot (c) in Figure 9.6. 

Plots of pairs of factor scores using estimated loadings from two solution 
methods are also good tools for detecting outliers. If the sets of loadings for a factor 
tend to agree, outliers will appear as points in the neighborhood of the 45 “ line btU 
far from the origin and the cluster of the remaining points. It is clear from Plot (b) m 
Figure 9 6 that one of the 276 observations is not consistent with the others. It has an 
SSLhZi. lyscore When .his potal, [39.1,303,75.7,115.73,4, »1], - 
removed and the analysis repeated, the loadings were not altered appreciably. 

When the data set is large, it should be divided into two (roughly) equal sets, 
and a factor analysis should be performed on each half The results of these analyses 
1 be compared with each other and with the analysis for the full data set to 
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-3-2-1012 3 

Figure 9.5 Factor scores for the first two factors of chicken-bone data. 


test the stability of the solution. If the results are consistent with one another, 
confidence in the solution is increased. 

The chicken-bone data were divided into two sets of n x = 137 and n 2 = 139 
observations, respectively. The resulting sample correlation matrices were 

“ 1.000 
.696 1.000 

.588 .540 1.000 

1 " .639 .575 .901 1.000 

.694 .606 .844 .835 1.000 

_ .660 .584 .866 .863 .931 1.000 

and 

~ 1.000 
.366 1.000 

.572 .352 1.000 

2 “ .587 .406 .950 1.000 

.587 .420 .909 .911 1.000 

.598 .386 .894 .927 .940 1.000 
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(a) First factor 

Figure 9.6 Pairs of factor scores for the chicken-bone data. (Loadings are 
estimated by principal component and maximum likelihood methods.) 


The rotated estimated loadings, specific variances, and proportion of the total 
(standardized) sample variance explained for a principal component solution of an 
m — 3 factor model are given in Table 9.11 on page 525. 

The results for the two halves of the chicken-bone measurements are very simi¬ 
lar. Factors F 2 and F\ interchange with respect to their labels, skull length and skull 
breadth, but they collectively seem to represent head size. The first factor, F *, again 
appears to be a body-size factor dominated by leg and wing dimensions. These are 
the same interpretations we gave to the results from a principal component factor 
analysis of the entire set of data. The solution is remarkably stable, and we can be 
fairly confident that the large loadings are “real.” As we have pointed out however, 
three factors are probably too many. A one- or two-factor model is surely sufficient 
for the chicken-bone data, and you are encouraged to repeat the analyses here with 
fewer factors and alternative solution methods. (See Exercise 9.10.) 
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(continued) 
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Table 9.1 1 


First set 

(«i = 137 observations) 
Rotated estimated factor loadings 

Second set 

(n 2 = 139 observations) 
Rotated estimated factor loadings 

Variable 

F\ F* 2 F\ 

fa 

F'l F\ F% & 

1. Skull length 

2. Skull breadth 

3. Femur length 

4. Tibia length 

5. Humerus length 

6 . Ulna length 

| 


■ 

.352 
.203 
( .930' 
.925 
.912 
U>14j 

C92P .167 .00 

■ 145 C968) .00 

.239 .130 .06 

.248 .187 .05 

.252 .208 .06 

.272 .168 .06 

Cumulative proportion 
of total (standardized) 
sample variance 
explained 

.546 .743 .940 

— 


•593 .780 .962 
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(c) Third factor 
Figure 9.6 ( continued ) 

Factor analysis has a tremendous intuitive appeal for the behavioral and social 
sciences. In these areas, it is natural to regard multivanate observat.ons on anima 
and human processes as manifestations of underlying unobservable traits. Fac 
analysis provides a way of explaining the observed variability in behavior 

° £ th StiU, whin all is said and done, factor analysis remains very subjective. Our exam¬ 
ples, in common with most published sources, consist of situations m which 
analysis model provides reasonable explanations in terms o a ew m e p 
tors. In practice, the vast majority of attempted factor analyses do no y ^ 

cut results. Unfortunately, the criterion for judging the quality o y 
has not been well quantified. Rather, that quality seems to depend on a 

WOW criterion 

If, while scrutinizing the factor analysis, the investigator can shout Wow, I under 
stand these factors” the application is deemed successful. 




Some Computational Details 
for Maximum Likelihood 
Estimation 


Although a simple analytical expression cannot be obtained for the maximum 
likelihood estimators L and they can be shown to satisfy certain equations. Not 
surprisingly, the conditions are stated in terms of the maximum likelihood estimator 

n 

S„ = (1/n) ^ (X; - X)(X ; - — X)' of an unstructured covariance matrix. Some 
j =i 

factor analysts employ the usual sample covariance S, but still use the title maximum 
likelihood to refer to resulting estimates. This modification, referenced in Footnote 4 
of this chapter, amounts to employing the likelihood obtained from the Wishart 

n 

distribution of (X ; * — X) (Xy — X)' and ignoring the minor contribution due to 

j-i _ 

the normal density for X. Hie factor analysis of R is, of course, unaffected by the 
choice of S„ or S, since they both produce the same correlation matrix. 

Result 9A.I. Let X!,x 2 ,... ,x„ be a random sample from a normal population. 
The maximum likelihood estimates L and V are obtained by maximizing (9-25) 
subject to the uniqueness condition in (9-26). They satisfy 

('H/ 2 S>- 1 / 2 ) (£- 1 / 2 L) = (^' ] / 2 L)(I + A) (9A-1) 

so the ;th column of ‘4 f-1 / 2 L is the (nonnormalized) eigenvector of '4 r ' 1 / 2 S n , 'i r ~ 1 / 2 
corresponding to eigenvalue 1 + A/. Here 

s„ = n~ l 2 ( x i - s) (Xj - x)' = n~ l (n - 1)S and A, > A 2 > ■ • • 5= A m 

7 —1 


527 




528 Chapter 9 Factor Analysis and Inference for Structured Covariance Matrices 


Also, at convergence, 

fa = ith diagonal element of S„ - LL' (9A-2) 

and 

tr(2 -1 S„) = p 

We avoid the details of the proof. However, it is evident that £ = x and a consideration 
of the log-likelihood leads to the maximization of -(n/ 2) [ln| X | + tr(X S„)] over L 
and Equivalently, since S„ and p are constant with respect to the maximization, we 

minimize 


= ln| X |—In|S„| + tr(2 'S„) - p (9A-3) 

subject to L"'F ! L — A, a diagonal matrix. ■ 


Comment. Lawley and Maxwell [10], along with many others who do factor 
analysis use the unbiased estimate S of the covariance matrix instead of the maxi¬ 
mum likelihood estimate S„. Now, (n - IjS has, for normal data, a Wishart distrib¬ 
ution. [See (4-21) and (4-23).] If we ignore the contribution to the likelihood in 
(9-25) from the second term involving (p - x), then maximizing the reduced likeli¬ 
hood over L and 'J' is equivalent to maximizing the Wishart likelihood 

Likelihood oc | 2 |-(«-.)/2 e -l(-i)/2] , ns-'s ] 


over L and Equivalently, we can minimize 

In | 2 | + tr(2 _1 S) 

or, as in (9A-3), 

ln|2|+ tr(2-*S)-ln|S|-p 


Under these conditions, Result (9A-1) holds with S in place of S„. Also, for large n, 
S and S„ are almost identical, and the corresponding maximum likelihood estimates, 
L and would be similar. For testing'the factor model [see (9-39)], |LL' + ¥ | 
should be compared with | S„ | if the actual likelihood of (9-25) is employed, and 
| LL' + ¥ | should be compared with | S | if the foregoing Wishart likelihood is used 

to derive L and 


Recommended Computational Scheme 


Form > 1 the condition L"^ _1 L= A effectively imposes m(m - 1 )/2 constraints 
on the elements of L and ¥, and the likelihood equations are solved, subject to 
these contraints, in an iterative fashion. One procedure is the following: 


1. 


Compute initial estimates of the specific variances </r,, fa, • • •, Jbreskog [8] 

suggests setting 



where s“ is the ith diagonal element of S 1 . 
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2. Given 'S', compute the first m distinct eigenvalues, A T > A 2 > “ • > A m > 1, and 
corresponding eigenvectors, e 1 } e 2 ,... ,e m , of the “uniqueness-rescaled” covari¬ 
ance matrix 

S* = ^ _ 1 / 2 S n ^“ 1/2 (9A-5) 

Let E = [ej i e 2 j • • • j e m ] be the p X m matrix of normalized eigenvectors 
and A = diag[A 1? A 2 ,...,A ffl ] be the m X m diagonal matrix of eigenvalues. 
From (9A-1), A = I + A and E = 'I ' -1 / 2 LA” 1 / 2 . Thus, we obtain the estimates 

L = ^’^EA 1 / 2 = ¥ 1 / 2 E(A - I ) 1/2 (9A-6) 

3. Substitute L obtained in (9A-6) into the likelihood function (9A-3), and 
minimize the result with respect to fa, tj/ 2 , ■.., A numerical search routine 
must be used. The values <Ai, 4*2* * • ■ obtained from this minimization are 
employed at Step (2) to create a new L. Steps (2) and (3) are repeated until con¬ 
vergence—that is, until the differences between successive values of and ipi 
are negligible. 

Comment. It often happens that the objective function in'(9A-3) has a relative 
minimum corresponding to negative values for some fa. This solution is clearly 
inadmissible and is said to be improper, or a Heywood case. For most packaged 
computer programs, negative fa, if they occur on a particular iteration, are changed 
to small positive numbers before proceeding with the next step. 

Maximum Likelihood Estimators of p = \. Z V Z + '¥ z 

When X has the factor analysis structure X — LL' + p can be factored as 
p = y-i/2xv-V2 = (V“ 1 / 2 L)(V“ 1 /2l)' + y-i/2^v _1/2 = L Z L^ + ¥ z . The loading 
matrix for the standardized variables is L z = V“ 1 // 2 L, and the corresponding specific 
variance matrix is >P Z = yV^V - !/ 2 , where \~ 1 / 2 is the diagonal matrix with ith 
diagonal element cr^ 2 . If R is substituted for S,j in the objective function of (9A-3), 
the investigator minimizes 

/il.l; + ¥.|\ , 

ln ]it| J + tr[(LlLi+1 ' ,) R| ‘ p 9A ' 7 

Introducing the diagonal matrix V 1/7 , whose ith diagonal element is the square 
root of the ith diagonal element of S n , we can write the objective function in (9A-7) as 

In(' ^ / 2 ^l LzL * + + tr[(L z L[ + ^J^V-^V^RV^V -1 / 2 ] - p 

\ Iv^iiriiv 1 / 2 ! J lV 

(\ (V^LOfV^L,)' + V'/z^v 1 / 2 |\ 

~' n { IS. | J 

+ tr[((V 1 / 2 L z )(V 1 / 2 L z )' + V 1 / 2 ^ z V’/ 2 )- 1 S n ] - p 

/|LL' + ¥|'\ .... - , 

“ ln ( j s „| J + tr[(LL ' + S »1 " P 


(9A-8) 
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The last inequality follows because the maximum likelihood estimates L and ^ 
minimize the objective function (9A-3). [Equality holds in (9A-8) for L x = 
and = V -1 / 2 ¥V _1/2 j Therefore, minimizing (9A-7)over L x and is equivalent 

to obtaining L and ¥ from S„ and estimating L z = V 1/2 L by L z = V 1/7 L and^ 
\p z = y~i/ 2 yjry-i /2 by ^ _ y-i/ 2 ^y-i /2 q^ e rationale for the latter procedure 
comes from the invariance property of maximum likelihood estimators. [See (4-20).] 


Exercises 


9.1. Show that the covariance matrix 

"1.0 *63 .45" 
p = .63 1.0 .35 

[.45 .35 1.0 _ 

for the p — 3 standardized random variables Zj, Z 2 , and Z 3 can be generated by the 
m — 1 factor model 

Z] = .9 Fj + ei 
Z 2 = JF l + e 2 
Z 3 = .5F { + 63 


where Var (Fi) = 1, Cov(e, F-[) — 0, and 


¥ = Cov(e) 


.19 0 0 
0 .51 0 

0 0 .71 


That is, write p in the form p = LL' + ¥ . 

9.2. Use the information in Exercise 9.1. 

(a) Calculate communalities hf, i = 1,2,3, and interpret these quantities. 

(b) Calculate Corr(Z„/ ? 1) for i = 1,2,3. Which variable might carry the greatest 
weight in “naming” the common factor? Why? 

9.3. The eigenvalues and eigenvectors of the correlation matrix p in Exercise 9.1 are 

A] - 1.96, el = [.625, .593, .507] 

A 2 = . 68 , e 2 = [-.219,-.491, .843] 

A 3 = .36, ej = [.749,-.638,-.177] 

(a) Assuming an m = 1 factor model, calculate the loading matrix L and matrix of 
specific variances ¥ using the principal component solution method. Compare the 
results with those in Exercise 9.1. 

(b) What proportion of the total population variance is explained by the first common factor? 

9.4. Given p and ¥ in Exercise 9.1 and an m = 1 factor model, calculate the reduced 
correlation matrix p = p — ¥ and the principal factor solution for the loading matrix L 
Is the result consistent with the information in Exercise 9.1? Should it be? 

9.5. Establish the inequality (9-19). 

Hint: Since S - LL' - ¥ has zeros on the diagonal, 

(sum of squared entries of S — LL' - $) < (sum of squared entries of S - LL 
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P; 2) , where P (2) = [e m+1 j- • i e p 


Now, S-LL' = A m+1 e m+1 e^+, + • • • + \ p e p t' p = P (2) A (2) P( 2) , whereP (2) = [e m+ ,j-•••«,] 
and A( 2 ) is the diagonal matrix with elements A m+1 ,..., \ p . 

Use (sum of squared entries of A) = trAA'and tr [P( 2 )A( 2 )A( 2 )Pj 2 )] =tr [ A (2 ) A ( 2 )]- 

9 . 6 . Verify the following matrix identities. 

(a) (I + L , ^~ 1 L)“ 1 L , ^' 1 L = I - (I + L'^L) -1 
Hint: Premultiply both sides by (I + L'^^L). 

(b) (LL' + ¥)"' = ’p-'Ltl + L"4 , ' 1 L) -I L"4 ,_1 

Hint: Postmultiply both sides by (LL' + ¥) and use (a). 

(c) L'(LL' + ¥)“* = (I + L'^-'LJ-'L'^- 1 

Hint: Postmultiply the result in (b) by L use (a), and take the transpose, noting that 
(LL' + ¥) 1 , ¥ -1 , and (I + L'^P^L) are symmetric matrices. 

9.7. (The factor model parameterization need not be unique.) Let the factor model with 
p - 2 and m = 1 prevail. Show that 


o"u ~ ^11 + ‘/'i 


a )2 ~ a 2 \ ~ ^ 11^2 


°22 = ^21 + < f > 2 

and, for given o\ \, a 22 , and 2 , there is an infinity of choices for L and ¥. 

9.8. (Unique but improper solution: Hey wood case.) 

Consider an m = 1 factor model for the population with covariance matrix 

“ 1 .4 .9" 

X = .4 1 .7 

.9 .7 1 


Show that there is a unique choice of L and ¥ with X = LL' 
the choice is not admissible. 


but that i/f 3 < 0 , so 


9.9. In a study of liquor preference in France, Stoetzel [14] collected preference rankings of 
p = 9 liquor types from n = 1442 individuals. A factor analysis of the 9 X 9 sample 
correlation matrix of rank orderings gave the following estimated loadings: 


Estimated factor loadings 


Variable (A - ,) 

F , 


*3 

Liquors 

.64 

.02 

.16 

Kirsch 

.50 

-.06 

-.10 

Mirabel le 

.46 

-.24 

-.19 

Rum 

.17 

.74 

.97* 

Marc 

-.29 

.66 

-.39 

Whiskey 

-.29 

-.08 

.09 

Calvados 

-.49 

.20 

-.04 

Cognac 

-.52 

-.03 

.42 

Armagnac 

-.60 

-.17 

.14 


*This figure is too high. It exceeds the maximum value of .64, as a result 
of an approximation method for obtaining the estimated factor loadings 
used by Stoetzel. 
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Given these results, Stoetzel concluded the following:The major principle of liquor pref¬ 
erence in France is the distinction between sweet and strong liquors. The second moti¬ 
vating element is price, which can be understood by remembering that liquor is both an 
expensive commodity and an item of conspicuous consumption. Except in the case of 
the two most popular and least expensive items (rum and marc), this second factor plays 
a much smaller role in producing preference judgments. The third factor concerns the 
sociological and primarily the regional, variability of the judgments. (See [14], p. 11.) 

(a) Given what you know about the various liquors involved, does Stoetzel’s interpreta¬ 
tion seem reasonable? 

(b) Plot the loading pairs for the first two factors. Conduct a graphical orthogonal rota¬ 
tion of the factor axes. Generate approximate rotated loadings. Interpret the rotated 
loadings for the first two factors. Does your interpretation agree with Stoetzel’s 
interpretation of these factors from the unrotated loadings? Explain. 

9.10. The correlation matrix for chicken-bone measurements (see Example 9.14) is 

' 1.000 

.505 1.000 

.569 .422 1.000 

.602 .467 .926 1.000 

.621 .482 . 877 .874 1.000 

.603 .450 .878 .894 .937 1.000_ 

The following estimated factor loadings were extracted by the maximum likelihood 
procedure: 


Variable 

Estimated 
factor loadings 

F, F 2 

Varimax 

rotated estimated 
factor loadings 

f ; f\ 

1 . Skull length 

.602 

.200 

.484 

.411 

2. Skull breadth 

.467 

.154 

.375 

.319 

3. Femur length 

.926 

.143 

.603 

.717 

4. Tibia length 

1.000 

.000 

319 

.855 

5. Humerus length 

.874 

.476 

.861 

.499 

6 . Ulna length 

.894 

.327 

.744 

.594 


Using the unrotated estimated factor loadings, obtain the maximum likelihood estimates 
of the following. 

(a) The specific variances. 

(b) The communalities. 

(c) The proportion of variance explained by each factor. 

(d) The residual matrix R - L r Li - ¥ z . 

9. * I . Refer to Exercise 9.10. Compute the value of the varimax criterion using both unrotated 
and rotated estimated factor loadings. Comment on the results. 

9.12. The covariance matrix for the logarithms of turtle measurements (see Example 8.4) is 


S = 10 


-3 


11.072 

8.019 6.417 
8.160 6.005 6.773 


^re f obtaT“ed maXimUm likelihood estimates ° f the factor loadings for 
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an m = 1 model 


Variable 

Estimated factor 
loadings 

Fi 

L ln(length) 

.2022 

2 . In(width) 

.0752 

3. In (height) 

.0765 


thefolbwtag ima,ed faCt ° r loadings ’ obtain the maximum likelihood estimates of each of 

(a) Specific variances. 

(b) Communalities. 

(c) Proportion of variance explained by the factor. 

(d) The residual matrix S n - LL' - 
Hint: Convert S to S n . 

9.!3. Refer to Exercise 9d2 Compute the test statistic in ( 9 - 39 ). Indicate why a test of 
this example. [L (9-40)V = ° ^ 2 UnreStricted cannot be carried ou, for 

9.14. The maximum likelihood factor loading estimates are given in (9A-6) by 

L = ¥ 1/2 EA ,/2 

Verify, for this choice, that 

L v-'L - a 

where A — A — I is a diagonal matrix. 

9 ‘ 15 ’ fccTuS e a a^d maJkeTvE 1 inVestigate * he consistency, determinants, and uses of 
anZ liri m ^ 6 mCaSUres of Profitability. As part of their study, a factor 

conduc ed The cori2J?£ “T?™? ^ market eStirnates of anomic profits was 
, C0rrelatl0n matnx of accounting historical, accounting replacement 

follows meaSUreS ° f pr ° fltability for a sam P‘ e of firms operating in 1977 is as 


___Variable 

Historical return on assets, HRA 
Historical return on equity, HRE 
Historical return on sales, HRS 
Replacement return on assets, RRA 
Replacement return on equity, RRE 
Replacement return on sales, RRS 
Market Q ratio, Q 
Market relative excess value, REV 


HRA HRE HRS RRA RRE RRS Q REV 

Flow ’ 1 

*738 1.000 

1 -731 .520 1.000 

.828 .688 .652 1.000 

•681 .831 .513 .887 1.000 

.712 .543 .826 .867 .692 1.000 

-625 322 .579 .639 .419 .608 1.000 

^■604 .303 .617 .563 .352 .610 .937 1.000 I 


534 Chapter 9 Factor Analysis and Inference for Structured Covariance Matrices 

The following rotated principal component estimates of factor loadings for an m = ; 
factor model were obtained: 



Estimated factor loadings 

Variable 

Fi 

Fi 


Historical return on assets 

.433 

.612 

,499 

Historical return on equity 

.125 

.892 

.234 

Historical return on sales 

.296 

.238 

.887 

Replacement return on assets 

.406 

.708 

.483 

Replacement return on equity 

.198 

.895 

.283 

Replacement return on sales 

.331 

.414 

.789 

Market Q ratio 

.928 

.160 

.294 

Market relative excess value 

.910 

.079 

.355 

Cumulative proportion 
of total variance explained 

.287 

.628 

.908 


(a) Using the estimated factor loadings, determine the specific variances and communalities. 

(b) Determine the residual matrix, R - L X L; - Given this information and the 
cumulative proportion of total variance explained in the preceding table, does an 
m - 3 factor model appear appropriate for these data? 

(c) Assuming that estimated loadings less than .4 are small, interpret the three factors. 

Does it appear, for example, that market-value measures provide evidence of 
profitability distinct from that provided by accounting measures? Can you sepa- 
rate accounting historical measures of profitability from accounting replacement 
measures? J 

9.16. Verify that factor scores constructed according to (9-50) have sample mean vector 0 and 
zero sample covariances. 

9.17. Refer to Example 9.12. Using the information in this example, evaluate (L'^LJ \ 

Note: Set the fourth diagonal element of to .01 so that can be determined. 
Will the regression and generalized least squares methods for constructing factors scores 
for standardized stock price observations give nearly the same results? Hint: See equation 
(9-57) and the discussion following it. 

The following exercises require the use of a computer. 

9.18. Refer to Exercise 8.16 concerning the numbers of fish caught. 

(a) Using only the measurements Xj — *4, obtain the principal component solution for 
factor models with m — 1 and m — 2. 

(b) Using only the measurements x, - x 4 , obtain the maximum likelihood solution for 
factor models with m — 1 and m = 2. 

(c) Rotate your solutions in Parts (a) and (b). Compare the solutions and comment on 
them. Interpret each factor. 

(d) Perform a factor analysis using the measurements x x - x 6 . Determine a reasonable^ 
number of factors m , and compare the principal component and maximum like 
hood solutions after rotation. Interpret the factors. 

9.19. A firm is attempting to evaluate the quality of its sales staff and is trying to find an ex¬ 
amination or series of tests that may reveal the potential for good performance in salesg 
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The firm has selected a random sample of 50 sales people and has evaluated each on 3 
measures of performance: growth of sales, profitability of sales, and new-account sales. 
These measures have been converted to a scale, on which 100 indicates “average” per¬ 
formance. Each of the 50 individuals took each of 4 tests, which purported to measure 
creativity, mechanical reasoning, abstract reasoning, and mathematical ability, respec¬ 
tively. The n - 50 observations on p = 1 variables are listed in Table 9.12 on page 536. 

(a) Assume an orthogonal factor model for the standardized variables Z,- = 
(Xi — pi)/ i ~ 1,2,..., 7. Obtain either the principal component solution or 
the maximum likelihood solution for m ~ 2 and m — 3 common factors. 

(b) Given your solution in (a), obtain the rotated loadings for m - 2 and m = 3. Com¬ 
pare the two sets of rotated loadings. Interpret the m —2 and m = 3 factor solutions. 

(c) List the estimated communalities, specific variances, and LL' + ¥ for the m = 2 
and m — 3 solutions. Compare the results. Which choice of m do you prefer at this 
point? Why? 

(d) Conduct a test of 7/ 0 : 2 — LL' + ¥ versus H Y : X ^ LL' + ¥ for both m - 2 and 
m = 3 at the a — .01 level. With these results and those in Parts b and c, which 
choice of m appears to be the best? 

(e) Suppose a new salesperson, selected at random, obtains the test scores x' = 
[jcj,jc 2 ,... ,* 7 ] =[110,98,105,15,18,12,35]. Calculate the salesperson’s factor 
score using the weighted least squares method and the regression method. 

Note: Hie components of x must be standardized using the sample means and vari¬ 
ances calculated from the original data. 

9.20. Using the air-pollution variables X ly X 2 ,X 5 , and X 6 given in Table 1.5, generate the 
sample covariance matrix. 

(a) Obtain the principal component solution to a factor model with m = 1 and m — 2. 

(b) Find the maximum likelihood estimates of L and ¥ for m = 1 and m = 2. 

(c) Compare the factorization obtained by the principal component and maximum like¬ 
lihood methods. 

9.21. Perform a varimax rotation of both m = 2 solutions in Exercise 9.20. Interpret the re¬ 
sults. Are the principal component and maximum likelihood solutions consistent with 
each other? 

9.22. Refer to Exercise 9.20. 

(a) Calculate the factor scores from the m ~ 2 maximum likelihood estimates by 
(i) weighted least squares in (9-50) and (ii) the regression approach of (9-58). 

(b) Find the factor scores from the principal component solution, using (9-51). 

(c) Compare the three sets of factor scores. 

9.23. Repeat Exercise 9.20, starting from the sample correlation matrix. Interpret the factors 
for the m = 1 and m = 2 solutions. Does it make a difference if R, rather than S, is 
factored? Explain. 

9.24. Perform a factor analysis of the census-tract data in Table 8.5. Start with R and obtain 
both the maximum likelihood and principal component solutions. Comment on your 
choice of m. Your analysis should include factor rotation and the computation of factor 
scores. 

9.25. Perform a factor analysis of the “stiffness” measurements given in Table 4.3 and dis¬ 
cussed in Example 4.14. Compute factor scores, and check for outliers in the data. Use 
the sample covariance matrix S. 
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9.26. Consider the mice-weight data in Example 8.6. Start with the sample covariance matrix. 
(See Exercise 8.15 for Vs^.) 

(a) Obtain the principal component solution to the factor model with m — 1 and 

m — 2, 

(b) Find the maximum likelihood estimates of the loadings and specific variances for 
m = 1 and m = 2. 

(c) Perform a varimax rotation of the solutions in Parts a and b. 

9.27. Repeat Exercise 9.26 by factoring R instead of the sample covariance matrix S. Also, for 
the mouse with standardized weights [.8, —.2, —.6,1.5], obtain the factor scores using 
the maximum likelihood estimates of the loadings and Equation (9-58). 

9.28. Perform a factor analysis of the national track records for women given in Table 1.9. Use 
the sample covariance matrix S and interpret the factors. Compute factor scores, and 
check for outliers in the data. Repeat the analysis with the sample correlation matrix R. 
Does it make a difference if R, rather than S, is factored? Explain. 

9.29. Refer to Exercise 9.28. Convert the national track records for women to speeds mea¬ 
sured in meters per second. (See Exercise 8.19.) Perform a factor analysis of the speed 
data. Use the sample covariance matrix S and interpret the factors. Compute factor 
scores, and check for outliers in the data. Repeat the analysis with the sample correlation 
matrix R. Does it make a difference if R, rather than S, is factored? Explain. Compare 
your results with the results in Exercise 9.28. Which analysis do you prefer? Why? 

9.30. Perform a factor analysis of the national track records for men given in Table 8.6. Repeat 
the steps given in Exercise 9.28. Is the appropriate factor model for the men’s data dif¬ 
ferent from the one for the women’s data? If not, are the interpretations of the factors 
roughly the same? If the models are different, explain the differences. 

9.31 . Refer to Exercise 9.30. Convert the national track records for men to speeds measured 
in meters per second. (See Exercise 8.21.) Perform a factor analysis of the speed data. 
Use the sample covariance matrix S and interpret the factors. Compute factor scores, 
and check for outliers in the data. Repeat the analysis with the sample correlation matrix 
R. Does it make a difference if R, rather than S, is factored? Explain. Compare your re¬ 
sults with the results in Exercise 9.30. Which analysis do you prefer? Why? 

9.32. Perform a factor analysis of the data on bulls given in Table 1.10. Use the seven variables 
YrHgt, FtFrBody, PrctFFB, Frame, BkFat, SaleHt, and SaleWt. Factor the sample covari¬ 
ance matrix S and interpret the factors. Compute factor scores, and check for outliers. 
Repeat the analysis with the sample correlation matrix R. Compare the results obtained 
from S with the results from R. Does it make a difference if R, rather than S, is factored? 
Explain. 

9.33. Perform a factor analysis of the psychological profile data in Bible 4.6. Use the sample 
correlation matrix R constructed from measurements on the five variables, Indep, Supp, 
Benev,Conform and Leader. Obtain both the principal component and maximum likeli¬ 
hood solutions for m — 2 and m = 3 factors. Can you interpret the factors? Your analy¬ 
sis should include factor rotation and the computation of factor scores. 

Note: Be aware that a maximum likelihood solution may result in a Heywood case. 

9.34. The pulp and paper properties data are given in Table 7.7. Perform a factor analysis 
using observations on the four paper property variables, BL, EM, SF, and BS and the 
sample correlation matrix R. Can the information in these data be summarized by a 
single factor? If so, can you interpret the factor? Try both the principal component and 
maximum likelihood solution methods. Repeat this analysis with the sample covariance 
matrix S. Does your interpretation of the factor(s) change if S rather than R is 
factored? 
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9.35. Repeat Exercise 9.34 using observations on the pulp fibeij*“* bleS AFL ' 
LFF, FFF, and ZST. Can these data be summarized by a single factor. Explai . 

9 36 Factor analyze the Mali family farm data in Table 8 . 7 . Use the sample correlation matrix 
TZ boThe Principal component and maximum likelihood solution methods for 
m = 3 4 and 5 factors Can you interpret the factors? Justify your choice of m. Your 
analysis should include factor rotation and the computation of factor scores. Can you 
identify any outliers in these data? 
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Canonical Correlation 
Analysis 


10.1 Introduction 

Canonical correlation analysis seeks to identify and quantify the associations 
between two sets of variables. H. Hotelling ([5], [ 6 ]), who initially developed 
the technique, provided the example of relating arithmetic speed and arithmetic 
power to reading speed and reading power. (See Exercise 10.9.) Other examples 
include relating governmental policy variables with economic goal variables and 
relating college “performance’* variables with precollege “achievement” variables. 

Canonical correlation analysis focuses on the correlation between a linear 
combination of the variables in one set and a linear combination of the variables in 
another set. The idea is first to determine the pair of linear combinations having 
the largest correlation. Next, we determine the pair of linear combinations having 
the largest correlation among all pairs uncorrelated with the initially selected pair, 
and so on. The pairs of linear combinations are called the canonical variables , and 
their correlations are called canonical correlations. 

The canonical correlations measure the strength of association between the two 
sets of variables. The maximization aspect of the technique represents an attempt to 
concentrate a high-dimensional relationship between two sets of variables into a 
few pairs of canonical variables. 


10.2 Canonical Variates and Canonical Correlations 


We shall be interested in measures of association between two groups of variables. 
The first group, of p variables, is represented by the (p X 1) random vector X^. The 
second group, of q variables, is represented by the X 1) random vector X (2 ^. We 
assume, in the theoretical development, that X^ represents the smaller set, so that 

P — q- 
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For the random vectors X (1) and X (2 \ let 

£(X (1) ) = m ( 1) ; Cov(X (1) ) = Xn 

£(X (2) ) = Cov(X (2) ) = X 2 2 (10-1) 

Cov(X0),X (2) ) = X 12 = X 2 i 

It will be convenient to consider X<» and X< 2 > jointly, so, using results (2-38) 
through (2-40) and (10-1), we find that the random vector 


X = 

((p + ^Xl) 


has mean vector 


, m _ fei 1 .!). 

M = £( X ) EYY(2h 

((/>+<7)Xl) L^ A > 


-ESQ 


and covariance matrix 


1 =£(X-m)(X-m)' 

(p+q)*(p+q) 


£(X (1 ) - i* (1) )(X (1 ) - |* 0) )' 
E(X& - m (2) )(X <V) - M (V) )' 


£(X (1) - m (1) )(X (2) - M (2) )' 
£(X< 


"in i 

Xu' 

(pxp) i 

(pX?) 

--f 

i 

^22 

_(<?xp) j 

(4 X ?) 


The covariances between pairs of variables from different sets—one variable 
from XW, one variable from X< 2 >—are contained in S 12 or, equivalently, in Sn- 
That is, the pq elements of l n measure the association between the two sets. When 
p and q are relatively large, interpreting the elements of X 12 collectively is ordinari¬ 
ly hopeless. Moreover, it is often linear combinations of variables that are interest¬ 
ing and useful for predictive or comparative purposes The mam task of canonical 
correlation analysis is to summarize the associations between the X an 
in terms of a few carefully chosen covariances (or correlations) rather than the pq 
covariances in S 12 . 
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Linear combinations provide simple summary measures of a set of variables. Set 

U = a'X^ 

V = b'X (2) (10-5) 

for some pair of coefficient vectors a and b.Then, using (10-5) and (2-45), we obtain 
Var (U) = a' Cov(X^)a = a'X n a 

Var(V) = b' Cov(X (2) )b = b'X 2 2 *> (10-6) 

Cov (U y V) = a' Cov(X^, X^)b = a'X^b 
We shall seek coefficient vectors a and b such that 


Corr (£/, V) 

is as large as possible. 

We define the following: 



(10-7) 


The first pair of canonical variables , or first canonical variate pair , is the pair of linear 
combinations C/ 1} having unit variances, which maximize the correlation (10-7); 

The second pair of canonical variables , or second canonical variate pair , is the pair 
of linear combinations t/ 2 , V 2 having unit variances, which maximize the correla¬ 
tion (10-7) among all choices that are uncorrelated with the first pair of canonical 
variables. 


At the kth step, 

The kth pair of canonical variables, or kth canonical variate pair , is the pair of 
linear combinations £4 > V* having unit variances, which maximize the correla¬ 
tion (10-7) among all choices uncorrelated with the previous k — 1 canonical 
variable pairs. 

The correlation between the Arth pair of canonical variables is called the kth canonical 
correlation . 

The following result gives the necessary details for obtaining the canonical 
variables and their correlations. 


Result 10.1. Suppose p < q and let the random vectors X^ and X (2) have 

(pxl) {qX 1) 

Cov(X (1) ) = X u ,Cov(X< 2 >) = I 22 andCov(X (1) ,X^) = I 12 , where 1 has full 

(P X P) (<7 X <7) (P x $) 

rank. For coefficient vectors a and b , form the linear combinations U = a'X^ 1J 

(P x l) (<?xi) 

and V = b r X< 2) .Then 

max Corr(£7, V) = p\ 

a,b 

attained by the linear combinations (first canonical variate pair) 

U } = ejS^X' 1 ’ and Vj = fiX^ /2 X (2) 
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The Ath pair of canonical variates, k = 2, 3,..., p, ' t 

U k = eiXlPxW V* = t'&FxW •/ 

maximizes 

Corr (U k , V k ) = p* k ~ 

among those linear combinations uncorrelated with the preceding 1,2,..., k - 
canonical variables. —- : 

Here p\ 2 > p? 2 • ■ • > p* 2 are the eigenvalues of Xn^nX^iXi} 72 , 

ej, e 2 .e p are the associated (p X 1) eigenvectors. [The quantities p\ 2 , p?,.. 

are also the p largest eigenvalues of the matrix X 2 PX 21 X 1 }Xi 2 ^ 22 ^ with correspon 
ing (q X 1) eigenvectors f 1( f 2 ,...,fp. Each f,- is proportional to 

The canonical variates have the properties j 

Var(I4) = Var(14) = 1 

Cov(U k ,U() = Corr (U k ,U t ) = 0 k * £ J 

Cov (V*. V t ) = Corr(14, V t ) = 0 k * ( ~t. 

Cov (U k , V t ) = Corr (U k , V t ) = 0 k * { 

forkj = 1,2,...,p. 

Proof. (See website: www.prenhall.com/statistics) , 


pjfev>> v- 


If the original variables are standardized with Z (1) = [Z^\ Z^\ • ■ •, Z*, 1 *]' an 
Z< 2 > = [Zi 2) , Z 2 2 \.... Zg ) ]\ from first principles, the canonical variates are of the form 

U k = a’ k zW = eipr^zW | 

V* = biZ (2) = fip^Z (2) (10-8 

Here,Cov(Zt*)) = p n , Cov(Z< 2 >) = p 22) Cov(Z<'),Z< 2 >) = pn = P 21 , ande, 
and f* are the eigenvectors of Pn 12 Pi 2 P 22 PiiPn' 7 and p/pPtiPi!P 12 P 22 ”. 
respectively. The canonical correlations, p*, satisfy 

Corr (17*, V k ) = p* k , k = 1,2.p (10-9 



pir P12P22P21P1 \ 
pnPiY 2 )' 


where p * 2 > P2 2 ^ • • • > p * 2 are the nonzero eigenvalues of the matrixIjjS 
pn^PnPjiPiiPu 12 (or, equivalently, the largest eigenvalues of Prf'PnPh 


Comment. Notice that 


ai(X<» - #»(»)) = a kl (X\ l> - /4 U ) + p* 2 (4’ ) " /4 1> ' 1 

+ --- + a * p (4 1) -p' 1 ’) 







^ Voqi a * 2 ^ ^2 


.(^ , -M? > ) 


Canonical Variates and Canonical Correlations 543 


where Var( A'/ ) = a ih i - 1 , 2 ,..., p. Therefore, the canonical coefficients for the 
standardized variables, Z- 1 * = (A^ - pi^)/Vo 7 i, are simply related to the canon¬ 
ical coefficients attached to the original variables AT* 1 ^. Specifically, if a* is the coeffi¬ 
cient vector for the /cth canonical variate U k , then a^V ^ 2 is the coefficient vector for 
the kth canonical variate constructed from the standardized variables Zt l \ Here V 
is the diagonal matrix with ith diagonal element Vcr^. Similarly, h k \is the coeffi¬ 
cient vector for the canonical variate constructed from the set of standardized vari- 
ab les Z^ 2 \ In this case V ^ 2 is the diagonal matrix with ith diagonal element Vcr“ = 

VVar (AT,- 2 *). The canonical correlations are unchanged by the standardization. 
However, the choice of the coefficient vectors a t , h k will not be unique if p ^ 2 = p ?+\. 

The relationship between the canonical coefficients of the standardized vari¬ 
ables and the canonical coefficients of the original variables follows from the special 
structure of the matrix [see also ( 10 - 11 )] 

Xii / 2 Xi 2 X 2 2X2iXii /2 or p\V 2 P12P22P21 P\V 2 

and, in this book, is unique to canonical correlation analysis. For example, in princi¬ 
pal component analysis, if a* is the coefficient vector for the kth principal compo¬ 
nent obtained from X, then a£.(X - fx) = a*V 1 / 2 Z, but we cannot infer that akV 1/2 
is the coefficient vector for the kth principal component derived from p . 

Example 10.1 (Calculating canonical variates and canonical correlations for stan¬ 
dardized variables) Suppose = [zj 1 ^,Z^]’ are standardized variables and 
Z( 2 > = [Z} 2 \ Z^]' are also standardized variables. Let Z = [Z (1 \ Z (2) ]' and 


Cov(Z) 


Pn. 

Pl 2 

.P21 

P 22 _ 


"1.0 

.4 

.5 

.6 

.4 

1.0 

.3 

.4 

.5 

.3 

1.0 

.2 

_ - 6 

A 

.2 

1.0 


1.0681 -.2229 

-.2229 1.0681 

1.0417 -.2083 
-.2083 1.0417 


P\M 2 P\lP 22 p 2 \P\]l ' 1 ~ 


.4371 .2178 
.2178 .1096 


The eigenvalues, pT^p?, of P 1 PP 12 P 22 P 21 P 1 K 2 are obtained from 


.4371 - A .2178 
.2178 .1096 - A 


(.4371 - A) (.1096 - A) - (2.178) 2 


A 2 - .5467A + .0005 
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yielding p \ 2 = .5458 and p? = .0009. The eigenvector ej follows from the vector 
equation 


.4371 .2178 
.2178 .1096 


ei = (.5458)6! 


Thus, e] = [.8947,-4466] and 


ai = Pn /2e i = 


From Result 10.1, fi « p^p 2 iPTl /2 ei and bj = Consequently, 

_j _ f.3959 .22921 [".85611 _ .4026 

bi oc p 22 P2i a i “ 5209 .3542_ _.2776_ [_.5443_ 


We must scale b! so that 


Var(Vi) = Var(b[Z (2 ') — b'!p 22 b! — 1 
The vector [.4026, .5443]' gives 


[.4026, .5443] 


1.0 .2 .4026 
.2 1.0 .5443 


= .5460 


Using V.5460 = .7389, we take 


1 P.4026 _ .5448 

bl ~ .7389 .5443 ~ [_.7366_ 

The first pair of canonical variates is 

Ux = aiZ (1) = .86ZI 1 ’ + ■28Z 2 1) 

Vi = b\Z m = .54Zf ) + ,74Z 2 2) 
and their canonical correlation is 

p\ = Vpf = V3458 - .74 

This is the largest correlation possible between linear combinations of variables 
from the Z (1) and Z (2) sets. 

The second canonical correlation, p 2 = V.0009 = .03, is very small, and conse¬ 
quently, the second pair of canonical variates, although uncorrelated with members of 
the first pair, conveys very little information about the association between sets (The 
calculation of the second pair of canonical variates is considered in Exercise 10.5.) 

We note that U\ and V\ , apart from a scale change, are not much different from 

the pair 

r 7 (in 

Ui = a'Z< 1 > = [3,1] % =3Z ( , 1) + zi 1) 


7 U] 

Vi = b'z< 2 > = [l, i] Zb) = zf ) + zf ) 
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For these variates, 

Var (t/j) = a'p n a = 12.4 
Var(T 1 ) = b'p 22 b = 2.4 
Cov(t/ 1 , Vj) = a p 12 b = 4.0 

and 

Corr(5 '- R) -^1^3-- 73 

The correlation between the rather simple and, perhaps, easily interpretable linear 
combinations U lt Vj is almost the maximum value p* — .74. ■ 

The procedure for obtaining the canonical variates presented in Result 10.1 has 
certain advantages. The symmetric matrices, whose eigenvectors determine the 
canonical coefficients, are readily handled by computer routines. Moreover, writing 
the coefficient vectors as a* = XfJ^e* and b* = X 22 ^f^ facilitates analytic descrip¬ 
tions and their geometric interpretations. To ease the computational burden, many 
people prefer to get the canonical correlations from the eigenvalue equation 

|sr!s 12 x^x 21 - P * 2 i| =o (io-io) 

The coefficient vectors a and b follow directly from the eigenvector equations 

Xi 1X125)22^21® = P* 2 ® 

5)225)2i5)i 1 Xi 2 b = p^b (10-11) 

The matrices Xi jXi 2 X225)2i and X 2 ]X 2 i5)ii 1 Xi2 are, in general, not symmetric. (See 
Exercise 10.4 for more details.) 

10.3 Interpreting the Population Canonical Variables 

Canonical variables are, in general, artificial. That is, they have no physical meaning. 
If the original variables X^ and X^ are used, the canonical coefficients a and b 
have units proportional to those of the X^ and X^ 2) sets. If the original variables 
are standardized to have zero means and unit variances, the canonical coefficients 
have no units of measurement, and they must be interpreted in terms of the stan¬ 
dardized variables. 

Result 10.1 gives the technical definitions of the canonical variables and canon¬ 
ical correlations. In this section, we concentrate on interpreting these quantities. 

Identifying the Canonical Variables 

Even though the canonical variables are artificial, they can often be “identified” 
in terms of the subject-matter variables Many times this identification is aided 
by computing the correlations between the canonical variates and the original 
variables. These correlations, however, must be interpreted with caution. They 
provide only univariate information, in the sense that they do not indicate how the 
original variables contribute jointly to the canonical analyses (See, for example, [11].) 
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Fbr this reason, many investigators prefer to assess the contributions of the original 
variables directly from the standardized coefficients (10-8). 

Let A = [a lt a 2 ,...,aJ' and B = [b 1 ,b z ,...,b 4 ] , < so that the vectors of 
(p*p) (qXq) 

canonical variables are 


U = AX (1) 

(pxl) 


V = bx (2) 
(^xl) 


( 10 - 12 ) 


where we are primarily interested in the first p canonical variables in V.Then 

Cov(U,X (1) ) = CovCAX^.xW) = A2„ (10-13) 

Because Var (U t ) = 1, Corr(l/;, A^ 1 ’) is obtained by dividing Cov(U h xi 0 ) by 

VVar (Af* 1 *) = cr}[k- Equivalently, Corr(fTj, A'*. 11 ) = Cov(f7,, o-j.j.'^A'j. *). Intro¬ 
ducing the (p X p) diagonal matrix V„ 1/2 with fcth diagonal element o-^J/ 2 , 
we have, in matrix terms, 

p ux o> = Corr(U,X(*)) = Cov(U, = Cov(AX(‘), Vi}/ 2 * 1 )) 

iPXP) = A2 n vri /2 

Similar calculations for the pairs (U, X (2) ), (V, X (2) ) and (V, X (1) ) yield 


p v x 0> = Aij 1V1 J'" Pv,x ,2) - ^^•22^22 2 

(pxpj (qxq) 

Pu,x (2) = ASiaViP Pv,x<" = BX 21 VTj /2 

(pXqj (qxp) 


(10-14) 


where V^ 72 is the (q X q) diagonal matrix with ith diagonal element [Var(A r ; ')]■ 
Canonical variables derived from standardized variables are sometimes inter¬ 
preted by computing the correlations. Thus, 

Pu.zCi = A t p n Pv,z <2) = P22 

Pu,zP) = A, P 12 Pv,z"» = B z P 21 (10-15) 

where A and B 2 are the matrices whose rows contain the canonical coefficients 
(pxp) (9x9) 

for the Z (1) and Z (2) sets, respectively. The correlations in the matrices displayed 
in (10-15) have the same numerical values as those appearing in (10-14); that is, 
Pv x“> = Puz< n . and s° forth. This follows because, for example, Pu,x (11 = 
A2 n Vn 1/2 = AV{fVri 1/2 2i 1 Vr, 1/2 = A.Pn = p VjZ m. The correlations are unaf- 
fected by the standardization. 


Example 10.2 (Computing correlations between canonical variates and their compo¬ 
nent variables) Compute the correlations between the first pair of canonical variates 
and their component variables for the situation considered in Example 10.1. 

The variables in Example 10.1 are already standardized, so equation (10-15) is 
applicable. For the standardized variables. 


1.0 .4 

Pu ~ .4 1.0 


1.0 .2 
.2 1.0 
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With p = 1, 


.86, .28] B 2 = [.54, .74] 


10 4 

Pu 1 ,z<» = A t p u = [.86,.28] 4 10 =[.97, .62] 


Pv„zP) = B I p 22 = [,54,.74] 10 2 ^ =[.69, .85] 

We conclude that, of the two variables in the set Z^, the first is most closely 
associated with the canonical variate U\. Of the two variables in the set Z (2 *, 
the second is most closely associated with V A . In this case, the correlations reinforce 
the information supplied by the standardized coefficients A 2 and B 2 . However, the 
correlations elevate the relative importance of Z 2 ^ in the first set and zj 2 ^ in 
the second set because they ignore the contribution of the remaining variable 
in each set. 

From (10-15), we also obtain the correlations 


Pt/i,z (2 ) ~ A z p l2 - [. 86 , . 28 ] 


[5 


Pv h z ( b - B z p 2 i - B 2 p; 2 - [.54, .74] 


= [.71, .46] 


Later, in our discussion of the sample canonical variates, we shall comment on 
the interpretation of these last correlations. ■ 


The correlations pu,x (1) an d Pv x (2) can help supply meanings for the canonical 
variates. The spirit is the same as in principal component analysis when the correla¬ 
tions between the principal components and their associated variables may provide 
subject-matter interpretations for the components. 


Canonical Correlations as Generalizations 
of Other Correlation Coefficients 

First, the canonical correlation generalizes the correlation between two variables. 
When X^ and each consist of a single variable, so that p = q = 1, 

|Corr(A'i 1) ,xi z) )| = | Corr (oY{ 1 \ bxi 2) )| for all a, b * 0 
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Therefore, the “canonical variates” U\ - and Vi — have correlation 
P* = |Corr(x[ 1) ,Xi 2J )|. When X* 1 ) and X* 2 ) have more components, setting 
a' = [0,..., 0,1, 0,.,., 0] with 1 in the ith position and b' = [0,..., 0,1, 0,..., 0] 
with 1 in the kth position yields 

|Corr(x} 1) ,xl 2) )| = |Corr(a'X (1) , b'X< 2 >)| 

< max CorrCa'X* 1 ), b'X< 2 >) = p\ (10 ' 16 ^ 

a,b 

That is, the first canonical correlation is larger than the absolute value of any entry 
inPn = VrJ^SnVjy 2 . 

Second, the multiple correlation coefficient p]( X ( 2 >) [see (7-48)] is a special case 
of a canonical correlation when X* 1 ) has the single element x[ 1 \p = 1). Recall that 

Pi(x (2) ) = max Corr (A^b'X* 2 )) = p\ for p = 1 (10-17) 

When p > 1, pi is larger than each of the multiple correlations of X- 1 ^ with X* 2 ) or 
the multiple correlations of with X* 1 ). 

Finally, we note that 

Pu k {x {2) ) = max Corr (t4,b'X* 2) ) = Corr(£4, V*) = pj, (10-18) 

k — 1,2,..., p 

from the proof of Result 10.1 (see website: www.prenhall.com/statistics). Similarly, 
Pv k (x^) = max Corr (a'X (1) , V k ) = Corr (U k ,V k ) = pj, (10-19) 

k = 1, 2,...,p 

That is, the canonical correlations are also the multiple correlation coefficients of U k 
with X* 2) or the multiple correlation coefficients of V k with X* 1 ). 

Because of its multiple correlation coefficient interpretation, the kth squared 
canonical correlation p * k 2 is the proportion of the variance of canonical variate U k 
“explained” by the set X* 2 ). It is also the proportion of the variance of canonical 
variate V k “explained” by the set X* 1 ). Therefore, p k is often called the shared vari¬ 
ance between the two sets X* 1 ) and X* 2) . The largest value, p* 2 , is sometimes regard¬ 
ed as a measure of set “overlap ” 


The First r Canonical Variables as a Summary of Variability 

The change of coordinates from X* 1} to U = AX (1) and from X* 2 > to V = BX (2) 
chosen to maximize Corr (f/j, V{] and, successively, Corr (U h V}), where (U h Vf) have 
zero correlation with the previous pairs (U\ n V x ) t (U 2 , V 2 ),..., (£^_j,V5_i). Cor 
relation between the sets X* 1 ) and X* 2) has been isolated in the pairs of canonical 
variables 

By design, the coefficient vectors a, , b, are selected to maximize correlations, 
not necessarily to provide variables that (approximately) account for the subset 
covariances X n and X 22 . When the first few pairs of canonical variables provide 
poor summaries of the variability in X] j and X 22 , it is not clear how a high canonical 
correlation should be interpreted. 
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Example 10.3 (Canonical correlation as a poor summary of variability) Consider the 
covariance matrix 



"100 

0 

0 

o' 

0 

1 

.95 

0 

0 

.95 

1 

0 

0 

0 

0 

100_ 


The reader may verify (see Exercise 10.1) that the first pair of canonical variates 
U\ — X^ and V 1 = X^ has correlation 

p\ = Corr (U u V x ) = .95 

Yet U\ = X 2 ^ provides a very poor summary of the variability in the first set. Most 
of the variability in this set is in X\ 1] , which is uncorrelated with £/, . The same situ¬ 
ation is true for Vi = X\ 2} in the second set. m 


A Geometrical Interpretation of the Population Canonical 
Correlation Analysis 

A geometrical interpretation of the procedure for selecting canonical variables 
provides some valuable insights into the nature of a canonical correlation analysis. 
The transformation 

U = AX* 1 * 

from X* 1 ) to U gives 

Cov(U) = AX n A' = I 

From Result 10.1 and (2-22), A = E'Xn /2 = E Pj Ai ltl P[ where E' is an orthogonal 
matrix with row e{, and = PjAjPJ. Now, P^X* 1 ) is the set of principal compo¬ 
nents derived from X (1) alone.The matrix A7 1/2 PiX* l) has ith row (1/ VXjp'X (1) , 
which is the ith principal component scaled to have unit variance. That is, 

Cov(Ar 1 / 2 PJX* 1 >) - Ar^P^nP^r 1 / 2 = 

= AJ^A.A^ 2 = I 

Consequently, U = AX* 1 ) = E'Pj A 1 ^PJX* 1 ) can be interpreted as (1) a 
transformation of X* 1 ) to uncorrelated standardized principal components, fol¬ 
lowed by (2) a rigid (orthogonal) rotation Pi determined by Xn and then (3) an¬ 
other rotation E' determined from the full covariance matrix X. A similar 
interpretation applies to V = BX* 2 ). 
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10.4 The Sample Canonical Variates and Sample 
Canonical Correlations 

A random sample of n observations on each of the (p + fi) variables X* ', X' 1 can 
be assembled into the n X (p + q) data matrix 

X = [x<*> i X< 2 >] 


( 10 - 20 ) 


r (1) • 

*12 

r (1) • 

*22 

. r (D 

*lp 

*2 p 

r (2) 

*11 

r (2) 

.*21 

v (2) . 
*12 

J 2 ) . 
*22 

r (2)n 
• x \q 

. 

*2 q 

Tx (1) ' 

x i 

v ( 2 )/ ~1 
x i 

J l ) . 
*;?2 

up 

r (2) 
x n 1 

v (2) . 
*«2 

H 

1 


x (2) ' 

J 


The vector of sample means can be organized as 

, ■ 


X 

(p+<7)Xl 


where x (1 * = — 2 x /^ 
n ;=i 

x< 2 >= 1 i x< 2) 


(10-21) 


Similarly, the sample covariance matrix can be arranged analogous to the represen¬ 
tation (10-4). Thus, 


(p+q)X(p+q) 


S11 

S ,2 ' 

(pxp) 

(px«) 

^21 

S22 

(<?Xp) 

(q*q) 


W11C1C 

n - 1 ;=1 

The linear combinations 

U = a'x (1) ; V = b'x (2) 
have sample correlation [see (3-36)] 

a ( S 12 b 

fU ' V Va'S„a 


k,l = 1,2 


( 10 - 22 ) 

(10-23) 

(10-24) 


The first pair of sample canonical variates is the pair of linear combinations 
U\, V\ having unit sample variances that maximize the ratio (10-24). 

In general, the kth pair of sample canonical variates is the pair of linear combinations 
U k , V k having unit sample variances that maximize the ratio (10-24) among those linear 
combinations uncorrelated with the previous k - 1 sample canonical variates. 

The sample correlation between U k and V k is called the kth sample canonical 

correlation. , 

The sample canonical variates and the sample canonical correlations can 
obtained from the sample covariance matrices S n , Si 2 = S 21 , and S 22 hi a manner 
consistent with the population case described in Result 10.1. 
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Result 10.2. Let p* 2 > p 2 2 ^ • ■ • > p* p 2 be the p ordered eigenvalues of 
Sri /2 Si 2 S 2 !S 2 iSn 1/2 with corresponding eigenvectors £ 1 , e 2 ,..., e p , where the S*/ are 
defined in (10-22) and p < q. Let f x , f 2 ,..., i p be the eigenvectors of SJ^S^Si} 
where the first p fs may be obtained from f* = (l/pJ)Sjy 2 S 2 iSry 2 e*> 
k = 1,2,..., p.Then the kth sample canonical variate pair 1 is 

Uk = ftSj-pxd) V k = fiS^xt 2 ) 

*1 bi 


where and x^ 2 ^ are the values of the variables X^ and for a particular 
experimental unit Also, the first sample canonical variate pair has the maximum 
sample correlation 


and for the £th pair, 


r UuV { “Pi 


r u k ,v k “ pi 


is the largest possible correlation among linear combinations uncorrelated with the 
preceding k - 1 samplecanonical variates. 

The quantities p*, p 2 > • • *, p* P are the sample canonical correlations. 2 


Proof. The proof of this result follows the proof of Result 10.1, with Ssubstituted 
for X kh k t l = 1,2. ■ 


The sample canonical variates have unit sample variances 

s u k ,u k = s v k ,v k - 1 (10-25) 

and their sample correlations are 

r u k ,u ( ~ r v k ,v t ~ ^ k ^ € 

r v k .v< = 0, k * € (10-26) 

The interpretation of U k , V k is often aided by computing the sample correlations be¬ 
tween the canonical variates and the variables in the sets and X (2) . We define 
the matrices 

A = [a,, a 2 ,..., a,]' B = [b 2 , b 2 ,..., b,]' (10-27) 

(pxp) (q'xq) 

whose rows are the coefficient vectors for the sample canonical variates. 3 Analogous 
to (10-12), we have 

U =Ax< 1} V = Bx^ (10-28) 

(pXl) (<?Xl) 


1 When the distribution is normal, the maximum likelihood method can be employed using X = S„ 
in place of S. The sample canonical correlations p k are, therefore, the maximum likelihood estimates of 
p m k and \/n/(n - 1) a*, \Zn/{n - 1) b* are the maximum likelihood estimates of a* and b* , respectively. 

2 If p > rank(Si2) = p \, the nonzero sample canonical correlations are p*,..., p* r 

3 The vectors bpi+1 - S22^fp,+l , bp, +2 — 822^^! +2. ■ • • > b 7 = are determined from a choice of 

the last <? - Pi mutually orthogonal eigenvectors f associated with the zero eigenvalue of ^ ‘ 
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and we can define 

R(j x a> = matrix of sample correlations of U with 
Rv z < 2 ) = matrix of sample correlations of V with x (2) 

Ru x (i) = matrix of sample correlations of LJ with x^ 

Ry x (i> = matrix of sample correlations of V with x^ 

Corresponding to (1049), we have 

Ru,x (l) ~ ASnDil^ 2 
Ry, x (2 ) = ®S22^22^ 

Ru,^» = ASuDsl 72 (10-29) 

R V, x (1) = BS21D! 1 / 2 

where is the (p X p) diagonal matrix with ith diagonal element (sample 

var(X' 1) )) _1/2 and D^ 2 is the (q X q) diagonal matrix with ith diagonal element 
(sample var(xp ) ))‘ 1/z . 

Comment. If the observations are standardized [see (8-25)], the data matrix 
becomes 


Z = [Z<*> | Z< 2 >] = 


and the sample canonical variates become 


V = B z z< 2 > 

(qXl) 


(10-30) 


where A z = ADff and B z = BD^ 2 - The sample canonical correlations are unaffect¬ 
ed by the standardization. The correlations displayed in (10-29) remain unchanged 
and may be calculated, for standardized observations, by substituting A z for 

A, B. for B, and R for S. Note that = I and D 2 1 /2 = I for standardized 
. .. (pXp) 

observations. 


Example 10.4 (Canonical correlation analysis of the chicken-bone data) In Example 
9.14, data consisting of bone and skull measurements of white leghorn fowl were 
described. From this example, the chicken-bone measurements for 


„ Hf*0>v skull length 

ea ' 1JV 2 '' = skull breadth 

i /v(2)t- I X? ] = femur length 

L*g(X<>). | X (2) = tibialength 
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have the sample correlation matrix 



1.0 

.505 

.569 

.602 

.505 

1.0 

.422 

.467 

.569 

.422 

1.0 

.926 

. .602 

.467 

.926 

1.0 


A canonical correlation analysis of the head and leg sets of variables 
using R produces the two canonical correlations and corresponding pairs of 
variables 


pi = .631 

and 

p| = .057 


U x = .781Z1 1 ’ + .345z^ 

Vi = .060z| 2) + .944zl 2) 

U 2 = —.856Z1 1 ’ + 1.106z^ 
V 2 = -2.648z| 2) + 2.475zf ) 


Here zP\ * = 1,2 and zp', i = 1,2 are the standardized data values for sets 1 and 
2, respectively. The preceding results were taken from the SAS statistical software 
output shown in Panel 10.1. In addition, the correlations of the original variables 
with the canonical variables are highlighted in that panel. m 


Example 10.5 (Canonical correlation analysis of job satisfaction) As part of a larger 
study of the effects of organizational structure on “job satisfaction,” Dunham [4] in¬ 
vestigated the extent to which measures of job satisfaction are related to job charac¬ 
teristics. Using a survey instrument, Dunham obtained measurements of p = 5 job 
characteristics and q = 7 job satisfaction variables for n = 784 executives from the 
corporate branch of a large retail merchandising corporation. Are measures of job 
satisfaction associated with job characteristics? The answer may have implications 
for job design. 


PANEL 10.1 SAS ANALYSIS FOR EXAMPLE 10.4 USING PROC CANCORR. 


title 'Canonical Correlation Analysis'; 
data skull (type = corr); 

_type_ = 'CORR'; 

input _name_$ xl x2 x3 x4; 

cards; 

xl 1.0 

x2 .505 1.0 

x3 .569 .422 1.0 

x4 .602 .467 .926 1.0 

PROGRAM COMMANDS 

proc cancorr data = skull vprefix = head wprefix = leg; 
var xl x2; with x3 x4; 



(continues on next page) 
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PANEL 10.1 (continued) 


Canonical Correlation Analysis 

- Adjusted Approx Squared 

Canonical Canonical Standard Canonical 

Correlation Correlation Error Correlation 


0.631085 

0.056794 


0.628291 


0.036286 0.398268 

0.060108 0.003226 


Raw Canonical Coefficient for the 'VAR' Variables 


HEAD1 

0.7807924389 

0.3445068301 


HEAD2 

-0.855973184 j 
1.1061835145 


Raw Canonical Coefficient for the 'WITH' Variables 


LEG1 

0.0602508775 

0.943948961 


LEG2 

-2.648156338 

2.4749388913 


Canonical Structure 

Correlations Between the 'VAR' Variables and Their Canonical Variables 


HEAD2 

-0.2974 

0.6739 


(see 10-29) 


Correlations Between the 'WITH' Variables and Their Canonical Variables 


LEG2 

-0.3564 

0.0227 


(see 10-29) 


Correlations Between the 'VAR' Variables 
and the Canonical Variables of the 'WITH' Variables 


LEG1 LEG2 

XI 0.6025 -0.0169 

X2 0.4663 0.0383 


(see 10-29) 


Correlations Between the 'WITH' Variables 
and the Canonical Variables of the 'VAR' Variables 


HEAD1 HEAD2 
X3 0.5897 -0.0202 

X4 0.6309 0.0013 


(see 10-29) 
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The original job characteristic variables, X^, and job satisfaction variables, 
x< 2 > , were respectively defined as 

x[^ feedback 

X 2 ^ task significance 

X* 1 ) = = task variety 

task identity 
_X^P _ _ autonomy 

supervisor satisfaction 
career-future satisfaction 
financial satisfaction 
workload satisfaction 
company identification 
kind-of-work-satisfaction 
general satisfaction 

Responses for variables X^ and X^ 2 ^ were recorded on a scale and then stan¬ 
dardized. The sample correlation matrix based on 784 responses is 


J: .5.12. 

1*21 | 


.0 





.33 

.32 

.20 

.19 

.30 

.37 

.21 

.49 

1.0 




.30 

.21 

.16 

.08 

.27 

.35 

.20 

.53 

.57 

1.0 



.31 

.23 

.14 

.07 

.24 

.37 

.18 

.49 

.46 

.48 

1.0 


.24 

.22 

.12 

.19 

.21 

.29 

.16 

.51 

.53 

.57 

.57 

1.0 

.38 

.32 

.17 

.23 

.32 

.36 

.27 

.33 

.30 

.31 

.24 

.38 

1.0 







.32 

.21 

.23 

.22 

.32 

.43 

1.0 






.20 

.16 

.14 

.12 

.17 

.27 

.33 

1.0 





.19 

.08 

.07 

.19 

.23 

.24 

.26 

.25 

1.0 




.30 

.27 

.24 

.21 

.32 

.34 

.54 

.46 

.28 

1.0 



.37 

.35 

.37 

.29 

.36 

.37 

.32 

.29 

.30 

.35 

1.0 


.21 

.20 

.18 

.16 

.27 

.40 

.58 

.45 

.27 

.59 

.31 

1.0 _ 


The min(p, q) = min(5,7) = 5 sample canonical correlations and the sample 
canonical variate coefficient vectors (from Dunham [4]) are displayed in the 
following table: 
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For example, the first sample canonical variate pair is 
t>! = .42z[ 1) + .21 + .17z^ - ,02z[ l) + Mzi 1 ' 1 

Vi = .42 z [ 2) + .22 z ( 2 2) - .03z3 V + -Olz^ + .29z ( s 2) + .52 z ( 6 2> - .12 z ( 7 2) 

with sample canonical correlation pf = .55. 

According to the coefficients, U\ is primarily a feedback and autonomy 
variable, while represents supervisor, career-future, and kind-of-work satisfaction, 
along with company identification. 

To provide interpretations for U\ andVj, the sample correlations between Ui 
and its component variables and between V x and its component variables were com¬ 
puted. Also, the following table shows the sample correlations between variables in 
one set and the first sample canonical variate of the other set. These correlations can 
be calculated using (10-29). 


Sample Correlations Between Original Variables and Canonical Variables 


variables 

Sample 

canonical 

variates 

variables 

Sample 

canonical 

variates 

tk 

Vi 

Ux 

vi 

1. Feedback 

.83 

.46 

1. Supervisor satisfaction 

.42 

.75 

2. Task significance 

.74 

.41 

2. Career-future satisfaction 

.35 

.65 

3. Task variety 

.75 

.42 

3. Financial satisfaction 

.21 

.39 

4. Task identity 

.62 

.34 

4. Workload satisfaction 

.21 

.37 

5. Autonomy 

.85 

.48 

5. Company identification 

.36 

.65 




6. Kind-of-work satisfaction 

.44 

.80 




7. General satisfaction 

.28 

.50 


All five job characteristic variables have roughly the same correlations with the 
first canonical variate U l . From this standpoint, U l might be interpreted as a job 
characteristic “index.” This differs from the preferred interpretation, based on 
coefficients, where the task variables are not important. 

The other member of the first canonical variate pair, Vi, seems to be represent¬ 
ing, primarily, supervisor satisfaction, career-future satisfaction, company identifica¬ 
tion, and kind-of-work satisfaction. As the variables suggest, V 1 might be regarded as 
a job satisfaction-company identification index. This agrees with the preceding 
interpretation based on the canonical coefficients of the z| 2 ^’s.The sample correla¬ 
tion between the two indices Ui and is p* = .55. There appears to be some over¬ 
lap between job characteristics and job satisfaction. We explore this issue further in 
Example 10.7. ■ 


Scatter plots of the first (Ui , Vi) pair may reveal atypical observations Xj requir¬ 
ing further study. If the canonical correlations p\, p%, ... are also moderately large, 
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scatter plots of the pairs (f/ 2 > V 2 ), (&3> ^ 3 )* • ♦ • ma y aIso be helpful in this rejL 
Many analysts suggest plotting “significant” canonical variates against their el l 
nent variables as an aid in subject-matter interpretation. These plots reinfofo 
correlation coefficients in (10-29). 

If the sample size is large, it is often desirable to split the sample in h; 
first half of the sample can be used to construct and evaluate the sample c, 
cal variates and canonical correlations. The results can then be “validate! 
the remaining observations. The change (if any) in the nature of the canL 
analysis will provide an indication of the sampling variability and the stabj| 
the conclusions. 


10.5 Additional Sample Descriptive Measures 


If the canonical variates are “good” summaries of their respective sets of varCT 
then the associations between variables can be described in terms of the.caj mag 
variates and their correlations. It is useful to have summary measures of the ext pi 
to which the canonical variates account for the variation in their respective setSjM 
also useful, on occasion, to calculate the proportion of variance in one set of vaga 
ables explained by the canonical variates of the other set. 


Matrices of Errors of Approximations 

Given the matrices A and B defined in (10-27), let and b (,) denote the ith col| 
of A” 1 and B - \ respectively. Since «J = Ax< 15 and V = Bx (2) we can write, jf 3 

I 


x (i) = A' 1 U 

(pXl) (pXp) [pxl) 


x (2) = B -1 V 
(?X1) (9x9) (9x1) 


Because sample Cov(U, V) = AS 12 B', sample Cov(U) = AS 11 A' - 
sample Cov(V) = BS 22 B' = 


h]2 


Pi 

0 •* 

• 0 


0 

Pi •• 

0 

0 





_ 0 

0 -■ 

• Pp 



(B- 1 )' = ^a^b' 1 )' + p 2 a (2) b (2) ' jj 

+ .-. + p p i^' jp 


Su = (A" 1 ) (A-*)' = + » (2)5<2) ' + - + ° (P) ° (P) ' 

S 22 = (IT 1 ) (IT 1 )' = b (1) b (1) ' + b (2) b (2) ' + •■• + b ( ’)b (9) ' 

Since x (1) = A' 1 11 and U has sample covariance I, the first r cola n^ 
contain the sample covariances of the first r canonical variates U u Ihj’gM 
their component variables X ( j\.. •, . Similarly, the first rcolu^ 

contain the sample covariances of Vi, V 2 ,. ■ ., K with ^ ie * r component v^JI 


Additional Sample Descriptive Measures 559 


If only the first r canonical pairs are used, so that for instance, 

x^) = [a^ j j ••• i a^] 


Ui 

U 2 


and 


x (2) = [b^ \ |... j b^j 


Ur] 

~vi 

V 7 

_K. 


(10-33) 


then Sj 2 is approximated by sample Cov (x* 1 ', x* 2) ). 

Continuing, we see that the matrices of errors of approximation are 

Su - (a* 1 ^ 1 )' + a (2) a (2) ' + ••• + a (r) a< r >') = a( r+I) a (r+1) ' + ••• + a^a^' 

S 22 - (b (I, b (1 >' + b (2) b (2) ' + •••+ b^b^') = b< r+I >b< r+1 >' + ••• + b (,) b (,) ' 

S 12 - (ria^b 11 )' + ^a( 2 >b< 2 )’ + ••• + ^a (r >b< r >') 

= ^ +1 a (r+1 )b( r+1) ' + ••• + P p a (p) b ip) ' 

(10-34) 

The approximation error matrices (10-34) may be interpreted as descriptive 
summaries of how well the first r sample canonical variates reproduce the sample 
covariance matrices. Patterns of large entries in the rows and/or columns of the ap¬ 
proximation error matrices indicate a poor “fit” to the corresponding variable(s). 

Ordinarily, the first r variates do a better job of reproducing the elements of 
Si 2 = S 2 j than the elements of Sn or S 2 2 - Mathematically, this occurs because the 
residual matrix in the former case is directly related to the smallest p — r sample 
canonical correlations. These correlations are usually all close to zero. On the other 
hand, the residual matrices associated with the approximations to the matrices Sj 1 and 
S 2 2 depend only on the last p - r and q - r coefficient vectors. The elements in 
these vectors may be relatively large, and hence, the residual matrices can have “large” 
entries. 

For standardized observations, R ki replaces S*/ and replace a w , 

in (10-34). 


Example 10.6 (Calculating matrices of errors of approximation) In Example 10.4, we 
obtained the canonical correlations between the two head and the two leg variables 
for white leghorn fowl. Starting with the sample correlation matrix 


®12 


'l.O 

.505 

.505 

1.0 

.569 

.422 

.602 

.467 

R 2 2_ 


.569 

.422 

1.0 

.926 



_ .602 

.467 

.926 

L0 
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we obtained the two sets of canonical correlations and variables 


= .631 


U 1 = .781 zi*’ + -345 
V, = ,060z < 1 2) + .944z2 2) 


_ t/ 2 = ~.856z[ 1) + 1.106Z2 1 ' 

P2 ‘ = 057 V 2 = —2.648zj 2) + 2.475z 2 2) 

where z f (1) , i = 1,2 and z?\i = 1,2 are the standardized data values for sets 1 and 
2 , respectively. 

We first calculate (see Panel 10.1) 


.781 .345 _1 _ .9548 -.2974 
-.856 1.106 “ .7388 .6739 


- , _ .9343 -.3564 

1 “ .9997 .0227 


Consequently, the matrices of errors of approximation created by using only the 
first canonical pair are 


R 12 — sampleCovf'zW, z^) = (.057) 5739 J t -0227] 


.006 -.000 
-.014 .001 


R n - sample Cov 


(z< 1) ) = 


[-.2974 .6739] 


.088 -.200 
-.200 -454 


R 22 — sample Cov(z* 2 *) 


[-.3564 .0227] 


.127 -.008 
-.008 .001 


where are given by ( 10 - 33 ) with r — 1 and aP, bP replace a , b 

respectively. 
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We see that the first pair of canonical variables effectively summarizes (repro¬ 
duces) the intraset correlations in R 12 . However, the individual variates are not 
particularly effective summaries of the sampling variability in the original z (1) and 
z^ sets, respectively. This is especially true for U \. ■ 

Proportions of Explained Sample Variance 

When the observations are standardized, the sample covariance matrices S*/ are 
correlation matrices R*/. The canonical coefficient vectors are the rows of the 
matrices A 2 and B 2 and the columns of Aj 1 and B' 1 are the sample correlations 
between the canonical variates and their component variables. 

Specifically, 

sample Cov (z^.U) = sample Cov (A; 1 U, U) = A' 1 

and 

sample Cov(z^, V) = sample Cov (B^V, V) — BJ 1 
so 





r U 2 ,z , \> ' 


A, 1 

= [ai 1) ,a< 2 »„. 

** g 

II 

.. 5 1 ' 

r 0 2 ,Z ( \> ' 

■' r 0 P A' 




r t>2.z'i> " 

• r £> P .z ( y 



r v l .z t ]> 

r V 2 ,z'‘,' 


BJ 1 

= tb< 1) ,bf)„. 

„bp] = V 

r v 2 ,fi " 

ru .rn 

V q^1 




^2.4’ 

r V 

q' z q . 


where and ry. ^ are the sample correlation coefficients between the quantities 
with subscripts. 

Using (10-32) with standardized observations, we obtain 
Total (standardized) sample variance in first set 

= tr(R n ) = tr(aP«P' + a‘ 2) a< 2) ' + ■ • • + a' p, a' p) ') = p (10-36a) 

Total (standardized) sample variance in second set 

= tr(R 22 ) = tr(b< 1) b< 1) ' + b< 2) b< 2) ' + • • • + b^bP') = q (10-36b) 

Since the correlations in the first r < p columns of A^ 1 and B" 1 involve only the 
sample canonical variates t/j, U 2t ..., U r and V x , V 2 > • ■ •, K> respectively, we define 
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Large Sample Inferences 563 


the contributions of the first r canonical variates to the total (standardized) samp| 
variances as 

+ a< 2) »' 2) ’ + ••• + W') = ± 

r=l *=1 Uh<k 

and 

trCb^bi 1 *' + b< 2) b‘ 2) ' + ••• + b< r) b<'>') = £ ± 4 ^, 

The proportions of total (standardized) sample variances “explained by” the fir# 
canonical variates then become 

( proportion of total standardized \ 
sample variance in first set 
explained by U u U 2 ,.. .,U r J 

_ tr(al 1) a< 1 >- + - + a< r) a< r) -) 

tf (^ll) 

2 2 r o„z<i> m 

= -£^-*=1- (10-37)Sl#i 

P 

and 

( proportion of total standardized \ 
sample variance in second set 
explained by 4L-4 j 

_ tr(bj 1) bl 1) / + ••■ + bj r) b‘ r) ') 
tr(R 22 ) 

2 2 r ^ I , z <2) 

= i=l £ = 1 p * 

q 

Descriptive measures (10-37) provide some indication of how well the canoni-^g 
cal variates represent their respective sets. They provide single-number descriptions ^ 
of the matrices of errors. In particular, 

±tr[R u - «’>' - £' 2) S' 2) '-W'] = 1 " 

— tr[R 22 - b^b' 0, - b‘ 2) b‘ 2) '-b' r) b' r) '] = 1 - R&\v h 0 2 V, 

according to (10-36) and (10-37). 


Example 10.7 (Calculating proportions of sample variance explained by canonical 
variates) Consider the job characteristic-job satisfaction data discussed in 
Example 10.5. Using the table of sample correlation coefficients presented in that 
example, we find that 

= 7 2 4 ,<;> = 7[(-83) 2 + (-74) 2 + - + (.85) 2 ] = .58 

Rfc |V, = ~i 4 = \ [(.75) 2 + (,65) 2 + - + (-50) 2 ] = .37 

The first sample canonical variate Uy of the job characteristics set accounts for 58% 
of the set’s total sample variance. The first sample canonical variate Vj of the job 
satisfaction set explains 37% of the set’s total sample variance. We might thus infer 
that 74 is a “better” representative of jts set than Vj is of its set. The interested read¬ 
er may wish to see how well U\ and Vj reproduce the correlation matrices R u and 
R 22 , respectively. [See (10-29).] ■ 


Large Sample Inferences 

When 24 2 = 0, a'X^ and b'X^ have covariance a'X^b = 0 for all vectors a and 
b. Consequently, all the canonical correlations must be zero, and there is no point in 
pursuing a canonical correlation analysis. The next result provides a way of testing 
X 12 — 0, for large samples. 

Result 10.3. Let 




xj ir 


Lxf’J 


j = 1,2, 


be a random sample from an N p+q (ji, X) population with 


Sn 

^12 

(P X P) 

(p x o) 

^21 

s 22 

(<l x p) 



Then the likelihood ratio test of H 0 : Xi 2 = 0 versus //j: 2 12 ^ 0 rejects H 0 for 

large values of (p x <*) (p x q) 


-2 In A = n In 


Sn 


322 


= -n hi n (1 “ ?i 




(10-38) 


«L 
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where 


S = 



S 12 


s 22 _ 


is the unbiased estimator of X. For large n , the test statistic (10-38) is approid 
distributed as a chi-square random variable with pq d.f. 


Proof. See Kshirsagar [8]. 

The likelihood ratio statistic (10-38) compares the sample generalized y 
under H Q , namely, 


Sn 0 

O' S 2 2 


= |Si 


with the unrestricted generalized variance | S |. 

Bartlett [3] suggests replacing the multiplicative factor n in the lik 
ratio statistic with the factor rc-l-^(p + <? + l)to improve the x 2 £ 
mation to the sampling distribution of -2 In A. Thus, for n and n — 
large, we 

Reject H 0 : X i2 = 0 (p\ = p 2 = • ■ • = p p = 0) at significance level a if 
-(/I - 1 - 1 (p + q + 1)) In n 0 - rf 2 ) > 
where xiq( a ) is the upper (100a)th percentile of a chi-square distribution! 


PQ 


d.f. 


If the null hypothesis H 0 : X 12 = 0 (p* = p 2 ~ * * ‘ = Pp ~ 0) is rejected, it 
ural to examine the “significance” of the individual canonical correlations. 
canonical correlations are ordered from the largest to the smallest, we can l 
assuming that the first canonical correlation is nonzero and the remaining 
canonical correlations are zero. If this hypothesis is rejected, we assume that t 
two canonical correlations are nonzero, but the remaining p ~ 2 canonical c 
tions are zero, and so forth. 

Let the implied sequence of hypotheses be 

H k 0 :p\ * 0* 0,..., P ;*0,p: + i=- = Pp = 0 


H\\p* * 0, for some i ^ k + 1 
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Bartlett [2] has argued that the &th hypothesis in (10-40) can be tested by the likeli¬ 
hood ratio criterion. Specifically, 

Reject Hq at significance level a if 

~(n - 1 - ~{p + q + l))ln 0 ~ P? 2 ) > X(p-k) (,-*)(“) (10-41) 

V Z / /=*+! 


where xf P -k){q-k)( a ) ls the upper (100a)th percentile of a chi-square distribution 
with (p — k)(q - k) d.f. We point out that the test statistic in (10-41) involves 

p _ 

n (1 — p* 2 ), the “residual” after the first k sample canonical correlations have 

/=*+i 

p _ 

been removed from the total criterion a 2/ - -no- p‘ 2 )- 

1=1 

If the members of the sequence H 0 , H^\ H q 2 \ and so forth, are tested one at 
a time until H ^ is not rejected for some k , the overall significance level is not a 
and, in fact, would be difficult to determine. Another defect of this procedure is the 
tendency it induces to conclude that a null hypothesis is correct simply because it is 
not rejected. 

To summarize, the overall test of significance in Result 10.3 is useful for multi¬ 
variate normal data. The sequential tests implied by (10-41) should be interpreted 
with caution and are, perhaps, best regarded as rough guides for selecting the num¬ 
ber of important canonical variates. 


Example 10.8 (Testing the significance of the canonical correlations for the job satis¬ 
faction data) Test the significance of the canonical correlations exhibited by the job 
characteristics-job satisfaction data introduced in Example 10.5. 

All the test statistics of immediate interest are summarized in the table on 
page 566. From Example 10.5, n = 784, p = 5, q = 7, p* = .55, p 2 = .23, p$ = .12, 
= .08, and = .05. 

Assuming multivariate normal data, we find that the first two canonical correla¬ 
tions, p\ and p 2 , appear to be nonzero, although with the very large sample size, 
small deviations from zero will show up as statistically significant. From a practical 
point of view, the second (and subsequent) sample canonical correlations can prob¬ 
ably be ignored, since (1) they are reasonably small in magnitude and (2) the corre¬ 
sponding canonical variates explain very little of the sample variation in the variable 
setsX (,) andX {2 >. ■ 


The distribution theory associated with the sample canonical correlations and 
the sample canonical variate coefficients is extremely complex (apart from the 
p — 1 and q = 1 situations), even in the null case,X 12 = O.The reader interested in 
the distribution theory is referred to Kshirsagar [8]. 
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Exercises 


10.1. Consider the covariance matrix given in Example 10.3: 



“100 

0 

0 

o' 

0 

1 

.95 

0 

0 

.95 

1 

0 

0 

0 

0 

0 

0 


Verify that the first pair of canonical variates are U\ = \ V\ = x[ 2 ^ with canonical 

correlation p* — .93. 

10.2. The (2X1) random vectors Xand X (2 ^ have the joint mean vector and joint covari¬ 
ance matrix 



(a) Calculate the canonical correlations p \, p 2 . 

(b) Determine the canonical variate pairs (C/j, V^) and (t/ 2 , V 2 ). 

(c) LetU = [U\, U 2 y and V = [V x , V 2 ]'. From first principles, evaluate 



and 



Compare your results with the properties in Result 10.1, 

10.3. LetZ^ = Vi'j/ 2 (X( 1 ) — and = V 22 ^ 2 (X^ — a*^) be two sets of standard¬ 

ized variables. If Pi, P 2 , - - ,P* P are the canonical correlations for the X^\ X^ sets and 
(U„v t ) = (a;x (I >,b!X< 2 >),i = 1 , 2 ,..., p, are the associated canonical variates, deter¬ 
mine the canonical correlations and canonical variates for the Z (1) ,Z^ sets.That is, 
express the canonical correlations and canonical variate coefficient vectors for the Z (1 \ 
Z^ sets in terms of those for the X^, X (2 ^ sets. 

10.4, (Alternative calculation of canonical correlations and variates 1 ) Show that, if A, is an 

eigenvalue of 2^22^21^1 P with associated eigenvector e*, then A t is also an 

eigenvalue of ^11^12^22^21 with eigenvector SiJ^e,. 

Hint: \ 2 ^ 22 ^ 21 ^ 11 /2 — A^I | = 0 implies that 

0 = 1 xn 1/2 11 xrK*Xi 2 S 5 iS 2 ,xrP - a ( i i i x}f i 

= I ^11^*12^22^21 — A,I | 
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10.5. 


10 . 6 . 


10.7. 


10 . 8 . 


Use the information in Example 10.1. rf 

(a) Find the eigenvalues of 27}2 12 XJ 2 2 2 1 and verify that these eigenvalues are thi 
same as the eigenvalues of Xn^i 2 X 22 X 21 X 11 ^. 

(b) Determine the second pair of canonical variates (U 2 , V 2 ) and verify, from first prind3 
pies, that their correlation is the second canonical correlation p 2 = .03. 

Show that the canonical correlations are invariant under nonsingular linear transforma¬ 
tions of the X (1 \ X (2) variables of the form C X^ and D X (2) . 

(pXp)</>xl) (qXq) {qXl) 


. Consider any linear combi 


\ cx C'i CX D' 

”dx’®’ J = I DX'^C'i Df 1? D' • Consider any linear combi 

nation a[(CX^^) = a'X^ with a' = aJC. Similarly, consider bJ(DX^) = b'X^ 
with b' = bJD. The choices aj = e'Xrl^C ' 1 and bj = give the maximum 

correlation. 


> _ \p P 
*\2 “ 

|_P Pj 


corresponding to the equal correlation 


structure where and X^ each have two components. 

(a) Determine the canonical variates corresponding to the nonzero canonical correlation. 

(b) Generalize the results in Part a to the case where X^ has p components and X^ 
has q > p components. 

Hint: p i2 = pH', where 1 is a (p X 1) column vector of l’s and 1' is a (q X 1) row 
vector of l’s. Note that p n l = [1 -I- (p - l)p]l so Pn /2 1 = [1 + (p ~ 1 )p]~ 1/r2 l. 

(Correlation for angular measurement.) Some observations, such as wind direction, are in 
the form of angles. An angle 8 2 can be represented as the pair X^ = [cos(0 2 ), sin(0 2 )]'. 

(a) Show that b'X* 2 ) = \/b\ + 6 2 cos(0 2 ~ P) where bf\/b\ + b 2 = cos(/3) and 
b 2 / y /b[ + b\ = sin(/3). 

Hint: cos(0 2 - p) ~ cos(0 2 ) cos(/3) + sin(0 2 ) sin(/3). 

(b) Let X^ 1 ) have a single component X^. Show that the single canonical correlation is 
pi = maxCorr (Af cos(0 2 — j3)). Selecting the canonical variable V\ amounts to 

0 

selecting a new origin p for the angle 6 2 . (See Johnson and Wehrly [7].) 

(c) Let X j 1 ^ be ozone (in parts per million) and 0 2 = wind direction measured from the 
north. Nineteen observations made in downtown Milwaukee, Wisconsin, give the 
sample correlation matrix 


R 11 

i*12 _ 

R 2. 

■ R 2 2_ 


ozone 

COS (0 2 ) 

sin (0 2 ) 

1.0 : 

.166 

.694 

.166 

1.0 

-.051 

.694 

-.051 

1.0 


Find the sample canonical correlation p* and the canonical variate V x representing 
the new origin p. '-?* 

(d) Suppose X (1) is also angular measurements of the form X (1 > = [cos (0J, sin (#i)] .y 
Thena'X^) = Va 2 + a 2 cos^ - «).Showthat J, 
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(e) Twenty-one observations on the 6:00 a.m. and noon wind directions give the correla¬ 
tion matrix 


10.9. 


cos($i) 
1.0 

-.291 

sin(6j) 
-.291 

1.0 

cos(0 2 ) 

.440 

-.205 

sin(6 2 ) 

.372 

.243 

.440 

-.205 

1.0 

.181 

.372 

.243 

.181 

1.0 


___ 





Find the sample canonical correlation p* and Ui,V } . 

The following exercises may require a computer. 

H. Hotelling [5] reports that n = 140 seventh-grade children received four tests 
on = reading speed, X 2 ^ ~ reading power, ~ arithmetic speed, and 

X 2 2 ^ = arithmetic power.The correlations for performance are 


R = 





’l.O 

.6328 

.2412 

Rn 

l R ! 2 


.6328 

1.0 

-.0553 

R 21 

IR 22 J 


.2412 

.0586 

-.0553 

.0655 

1.0 

.4248 


.0586 

.0655 

"4248 

1.0 


(a) Find all the sample canonical correlations and the sample canonical variates. 

(b) Stating any assumptions you make, test the hypotheses 

Hq'-Xi 2 = P 12 = 0 (pi = P 2 = 0) 

Hi : X \ 2 ~ P12 * 0 

at the a = .05 level of significance. If H 0 is rejected, test 

Pj 0,p 2 = 0 

H l P:A* 0 

with a significance level of a = .05. Does reading ability (as measured by the two 
tests) correlate with arithmetic ability (as measured by the two tests)? Discuss. 

(c) Evaluate the matrices of approximation errors for R x ! , R 22 , and R j 2 determined by 
the first sample canonical variate pair , V Y . 

10.10. In a study of poverty, crime, and deterrence, Parker and Smith [10] report certain sum¬ 
mary crime statistics in various states for the years 1970 and 1973. A portion of their 
sample correlation matrix is 



" 1.0 

.615 

-.111 

-.266 

.615 

1.0 

-.195 

-.085 

-.111 

-.195 

1.0 

-.269 

— .266 

-.085 

-.269 

1.0 


The variables are 

x[^ = 1973 nonprimary homicides 

X 2 ^ = 1973 primary homicides (homicides involving family or acquaintances) 

(2) 

X\ = 1970 severity of punishment (median months served) 

X\ = 1970 certainty of punishment (number of admissions to prison divided by 
number of homicides) 
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(a) Find the sample canonical correlations. ~ 

(b) Determine the first canonical pair f/j, V\ and interpret these quantities. 

I o.11. Example 8.5 presents the correlation matrix obtained from n = 103 success!! 
weekly rates of return for five stocks. Perform a canonical correlation analysis w2T 
X (a) (b) (c) (d) (e) * * * * * * (1) = [Ai*\ X[ l \ A^]', the rates of return for the banks, and X {2) = [a[ 2) , 
the rates of return for the oil companies. Jj| 

10 . 12 . A random sample of n ~ 70 families will be surveyed to determine the associatl! 
between certain “demographic” variables and certain “consumption” variables. 

Let . j 


Criterion 

set 

Predictor 

set 


X\ ] ~ annual frequency of dining at a restaurant 
X 2 1} = annual frequency of attending movies 

x [ 2) = age of head of household 

X ^ — annual family income 

( 2 ) 

X\ - educational level of head of household 


Suppose 70 observations on the preceding variables give the sample correlation matri 


Rn 

: R 12 

I<21 

;R 2 2_ 


"l.O 

.80 

1.0 


.26 

.33 

1.0 


.67 

.59 

.37 

1.0 

.34 

.34 

.21 

.35 1.0 


(a) Determine the sample canonical correlations, and test the hypothesis H 0 :2 
(or, equivalently, p x 2 ~ 0) at the a = .05 level. If H 0 is rejected, test for the signifi 
cance (a = .05) of the first canonical correlation. 

(b) Using standardized variables, construct the canonical variates corresponding to the |gj 

“significant” canonical correlation(s). jf|l 

(c) Using the results in Parts a and b, prepare a table showing the canonical variate co- rS: 

efficients (for “significant” canonical correlations) and the sample correlations ofy||f 
the canonical variates with their component variables. J 

(d) Given the information in (c), interpret the canonical variates 

(e) Do the demographic variables have something to say about the consumption vari-Jpg 

ables? Do the consumption variables provide much information about the demo- JpjgJ 

graphic variables? 

10.13. Waugh [12] provides information about n = 138 samples of Canadian hard red springs 

wheat and the flour made from the samples The p - 5 wheat measurements (in stair 

dardized form) were 

Z\^ = kernel texture 
~ test weight 
= damaged kernels 
— foreign material 

Z 5 ^ = crude protein in the wheat 
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The q = 4 (standardized) flour measurements were 

= wheat per barrel of flour 

( 2 ) 

z\ = ash in flour 

Z 3 = crude protein in flour 

zi 2) = gluten quality index 


The sample correlation matrix was 



1.0 


.754 

-.690 

-.446 

.692 

1.0 

-.712 

-.515 

.412 

1.0 

.323* 

-.444 

1.0 

-.334 

-.605 

-.722 

.737 

.527 

-.479 

-.419 

.361 

.461 

.780 

.542 

-.546 

-.393 

-.152 

-.102 

.172 

-.019 


1.0 


-.383 

1.0 



-.505 

.251 

1.0 


.737 

-.490 

-.434 

1.0 

-.148 

.250 

-.079 

-.163 



(a) Find the sample canonical variates corresponding to significant (at the a = .01 
level) canonical correlations 

(b) Interpret the first sample canonical variates C/ 7 , . Do they in some sense represent 

the overall quality of the wheat and flour, respectively? 

(c) What proportion of the total sample variance of the first set is explained by the 
canonical variate T/j? What proportion of the total sample variance of the Z (2) set is 
explained by the canonical variate Vj ? Discuss your answers. 

10.14. Consider the correlation matrix of profitability measures given in Exercise 9.15. Let X (1) 
= [A^, X ^\..., Ag 1 ^]' be the vector of variables representing accounting measures 
of profitability, and let X^ = [x{ 2 \ X^Y be the vector of variables representing the 
two market measures of profitability. Partition the sample correlation matrix accordingly, 
and perform a canonical correlation analysis. Specifically, 

(a) Determine the first sample canonical variates U x , V { and their correlation. Interpret 
these canonical variates. 

(b) Let Z W and Z^ be the sets of standardized variables corresponding to X^ and X^ 2 \ 
respectively. What proportion of the total sample variance of ZW is explained by 
the canonical variate U\? What proportion of the total sample variance of is 
explained by the canonical variate V^? Discuss your answers. 


10.15. Observations on four measures of stiffness are given in Table 4.3 and discussed in Exam¬ 
ple 4.14. Use the data in the table to construct the sample covariance matrix S. Let X^ 
= [A^, A^]' be the vector of variables representing the dynamic measures of stiffness 
(shock wave, vibration), and let X^ 2 ^ = [x[ 2 \ X^]' be the vector of variables represent¬ 
ing the static measures of stiffness. Perform a canonical correlation analysis of these data. 
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10.16. Andrews and Herzberg [1] give data obtained from a study of a comparison of nonil 
be tic and diabetic patients. Three primary variables, 

A'i 1 ^ = glucose intolerance 

Xp = insulin response to oral glucose 

= insulin resistance 

and two secondary variables, 

f 2) 

X\ — relative weight. 
xP = fasting plasma glucose 
were measured. The data for n - 46 nondiabetic patients yield the covariance mal 


10.17. 


S - 


"Sn |S U 1 = 
_^ 2 -l I S22J 


"1106.000 

396.700 

108.400 

.787 

26.230 

396.700 

2382.000 

1143.000 

-.214 

-23.960 

108.400 

1143.000 

2136.000 

2.189 

-20.840 

.787 

-.214 

2.189 

.016 

.216 

26.230 

-23.960 

-20.840 

.216 

70.560 


Determine the sample canonical variates and their correlations Interpret these quantities^ 
Are the first canonical variates good summary measures of their respective sets of vaj 
ables? Explain.Test for the significance of the canonical relations with a = .05. 

Data concerning a person’s desire to smoke and psychological and physical state 1 
collected for n = 110 subjects.The data were responses,coded 1 to 5, to each of 12 qii| 
tions (variables).The four standardized measurements related to the desire to smoke a 
defined as 

z[ ^ = smoking 1 (first wording) 
zp - smoking 2 (second wording) 
zP = smoking 3 (third wording) 
zP = smoking 4 (fourth wording) 

The eight standardized measurements related to the psychological and physical state a 
given by 


zP = concentration 

')mm 

(2) 

Z 2 = annoyance 


zP = sleepiness 


zP = tenseness 


zP = alertness 


zP = irritability 


zP = tiredness 


(2) 

Zg = contentedness 



The correlation matrix constructed from the data is 


Rn 

1*12 


S R 22_ 


where 



1.000 

.785 

.810 

.775 





.785 

1.000 

.816 

. .813 





.810 

.816 

1.000 

.845 





.775 

.813 

.845 

1.000_ 






.086 

.144 

140 .222 .101 

.189 

.199 

.239" 

» t 

.200 

.119 .211 .301 .223 

.221 

.274 

.235 

“21 ~ 

.041 

.060 . 

126 .120 .039 

.108 

.139 

.100 


.228 

.122 .277 .214 .201 

.156 

.271 

.171 _ 

1.000 

.562 

.457 

.579 

.802 

.595 

.512 

.492 

.562 

1.000 

.360 

.705 

.578 

.796 

.413 

.739 

.457 

.360 

1.000 

.273 

.606 

.337 

.798 

.240 

.579 

.705 

.273 

1.000 

.594 

.725 

.364 

.711 

.802 

.578 

.606 

.594 

1.000 

.605 

.698 

.605 

.595 

.796 

.337 

.725 

.605 

1.000 

.428 

.697 

.512 

.413 

.798 

.364 

.698 

.428 

1.000 

.394 

.492 

.739 

.240 

.711 

.605 

.697 

.394 

1.000 



Determine the sample canonical variates and their correlations. Interpret these quanti¬ 
ties. Are the first canonical variates good summary measures of their respective sets of 
variables? Explain. 

10.18. The data in Thble 7.7 contain measurements on characteristics of pulp fibers and the 
paper made from them. To correspond with the notation in this chapter, let the paper 
characteristics be 

jc^ = breaking length 
xP = elastic modulus 
= stress at failure 
= burst strength 

and the pulp fiber characteristics be 

Jtj 2) = arithmetic fiber length 
x^p = long fiber fraction 
= fine fiber fraction 
= zero span tensile 

Determine the sample canonical variates and their correlations. Are the first canonical 
variates good summary measures of their respective sets of variables? Explain. Tfest for 
the significance of the canonical relations with a = .05. Interpret the significant canoni¬ 
cal variables. 

10.19. Refer to the correlation matrix for the Olympic decathlon results in Example 9.6. Obtain 
the canonical correlations between the results for the running speed events (100-meter 
run, 400-meter run, long jump) and the arm strength events (discus, javelin, shot put). 
Recall that the signs of standardized running events values were reversed so that large 
scores are best for all events. 
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Chapter 



Discrimination and Classification 


II.I Introduction 

Discrimination and classification are multivariate techniques concerned with 
separating distinct sets of objects (or observations) and with allocating new objects 
(observations) to previously defined groups. Discriminant analysis is rather 
exploratory in nature. As a separative procedure, it is often employed on a one-time 
basis in order to investigate observed differences when causal relationships are not 
well understood. Classification procedures are less exploratory in the sense that 
they lead to well-defined rules, which can be used for assigning new objects. Classi¬ 
fication ordinarily requires more problem structure than discrimination does. 

Thus, the immediate goals of discrimination and classification, respectively, are 
as follows: 

Goal 1. To describe, either graphically (in three or fewer dimensions) or alge¬ 
braically, the differential features of objects (observations) from sever¬ 
al known collections (populations). We try to find “discriminants” 
whose numerical values are such that the collections are separated as 
much as possible. 

Goal 2. To sort objects (observations) into two or more labeled classes. The em¬ 
phasis is on deriving a rule that can be used to optimally assign new ob- 
, jects to the labeled classes. 

We shall follow convention and use the term discrimination to refer to Goal 2. 
This terminology was introduced by R. A. Fisher [10] in the first modem treatment 
of separative problems. A more descriptive term for this goal, however, is separa¬ 
tion. We shall refer to the second goal as classification or allocation. 

A function that separates objects may sometimes serve as an allocator, and, 
conversely, a rule that allocates objects may suggest a discriminatory procedure. In 
practice, Goals 1 and 2 frequently overlap, and the distinction between separation 
and allocation becomes blurred. 
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11.2 Separation and Classification for Two Populations 

To fix ideas, let us list situations in which one may be interested in (1) separating two 
classes of objects or (2) assigning a new object to one of two classes (or both). It is 
convenient to label the classes 77 r 1 and 7 t 2 . The objects are ordinarily separated or" 
classified on the basis of measurements on, for instance, p associated random vari¬ 
ables X' = [X x , x 2 ,..., X p \. The observed values of X differ to some extent from^ 
one class to the other. 1 We can think of the totality of values from the first class as 
being the population of x values for 77 ^ and those from the second class as the popu¬ 
lation of x values for 77 ^. These two populations can then be described by probabili¬ 
ty density functions f x (x) and / 2 (x), and consequently, we can talk of assigning 
observations to populations or objects to classes interchangeably. 

You may recall that some of the examples of the following separation- 
classification situations were introduced in Chapter 1. 


Populations 7Ti and 7 t 2 


Measured variables X 


1. Solvent and distressed property-liability 
insurance companies. 

2. Nonulcer dyspeptics (those with upset 
stomach problems) and controls 
(“normal”). 

3. Federalist Papers written by James 
Madison and those written by 
Alexander Hamilton. 

4. Two species of chickweed. 

5. Purchasers of a new product and 
laggards (those “slow” to purchase). 

6. Successful or unsuccessful (fail to 
graduate) college students. 

7. Males and females. 

8. Good and poor credit risks. 

9. Alcoholics and nonalcoholics. 


Tbtal assets, cost of stocks and bonds, market 
value of stocks and bonds, loss expenses, 
surplus, amount of premiums written. 

Measures of anxiety, dependence, guilt, 
perfectionism. 

Frequencies of different words and lengths of 
sentences. 

Sepal and petal length, petal cleft depth, bract 
length, scarious tip length, pollen diameter. 
Education, income, family size, amount of 
previous brand switching. 

Entrance examination scores, high school grade- 
point average, number of high school activities. 

Anthropological measurements, like 
circumference and volume on ancient skulls. 
Income, age, number of credit cards, family size. 
Activity of monoamine oxidase enzyme, activity 
of adenylate cyclase enzyme___ 


We see from item 5, for example, that objects (consumers) are to be separated 
into two labeled classes (“purchasers” and “laggards”) on the basis of observe 
values of presumably relevant variables (education, income, and so fort ). n e 
terminology of observation and population , we want to identify an observation or 

l If the values of X were not very different for objects in tt, and tt 2 , there would be n0 P rob *f^’ 
that is, the classes would be indistinguishable, and new objects could be assigned to ei er 
indiscriminately. 
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the form x' = [.^(education), x 2 (income), x 3 (familysize), x 4 (amount of brand 
switching).] as population 7r\, purchasers, or population 7r 2 , laggards. 

At this point, we shall concentrate on classification for two populations, return¬ 
ing to separation in Section 11.3. 

Allocation or classification rules are usually developed from “learning” sam¬ 
ples. Measured characteristics of randomly selected objects known to come from 
each of the two populations are examined for differences. Essentially, the set of all 
possible sample outcomes is divided into two regions, Ri and R 2 , such that if a new 
observation falls in it is allocated to population 7 t { , and if it falls in R 2 , we allo¬ 
cate it to population 7r 2 . Thus, one set of observed values favors 77 ^, while the other 
set of values favors 7 t 2 . 

You may wonder at this point how it is we know that some observations belong 
to a particular population, but we are unsure about others. (This, of course, is what 
makes classification a problem!) Several conditions can give rise to this apparent 
anomaly (see [20]): 

1. Incomplete knowledge of future performance. 

Examples: In the past, extreme values of certain financial variables were ob¬ 
served 2 years prior to a firm’s subsequent bankruptcy. Classifying another firm 
as sound or distressed on the basis of observed values of these leading indicators 
may allow the officers to take corrective action, if necessary, before it is too late. 

A medical school applications office might want to classify an applicant as 
likely to become M.D. or unlikely to become M.D. on the basis of test scores and 
other college records. Here the actual determination can be made only at the 
end of several years of training. 

2. " Perfect ” information requires destroy ing the object. 

Example: The lifetime of a calculator battery is determined by using it until 
it fails, and the strength of a piece of lumber is obtained by loading it until it 
breaks. Failed products cannot be sold. One would like to classify products as 
good or bad (not meeting specifications) on the basis of certain preliminary 
measurements. 

3. Unavailable or expensive information. 

Examples: It is assumed that certain of the Federalist Papers were written by 
James Madison or Alexander Hamilton because they signed them. Others of the 
Papers , however, were unsigned and it is of interest to determine which of the 
two men wrote the unsigned Papers. Clearly, we cannot ask them. Word fre¬ 
quencies and sentence lengths may help classify the disputed Papers. 

Many medical problems can be identified conclusively only by conducting 
an expensive operation. Usually, one would like to diagnose an illness from eas¬ 
ily observed, yet potentially fallible, external symptoms. This approach helps 
avoid needless—and expensive—operations. 

It should be clear from these examples that classification rules cannot usually 
provide an error-free method of assignment. This is because there may not be a 
clear distinction between the measured characteristics of the populations; that is, 
the groups may overlap. It is then possible, for example, to incorrectly classify a 7 t 2 
object as belonging to tt x or a 7 rj object as belonging to 7 t 2 . 
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Example II.I (Discriminating owners from nonowners of riding mowers) Consider 
two groups in a city: tt x , riding-mower owners, and 7 r 2 , those without riding mowers— 
that is, nonowners. In order to identify the best sales prospects for an intensive sales 
campaign, a riding-mower manufacturer is interested in classifying families as^ 
prospective owners or nonowners on the basis of jtj = income and x 2 - lot size. 
Random samples of «i = 12 current owners and n 2 = 12 current nonowners yield 
the values in Table 11.1. 



These data are plotted in Figure 11.1. We see that riding-mower owners tend to 
have larger incomes and bigger lots than nonowners, although income seems to be a 
better “discriminator” than lot size. On the other hand, there is some overlap be¬ 
tween the two groups. If, for example, we were to allocate those values of (^ 1 ,^ 2 ) 
that fall into region R 1 (as determined by the solid line in the figure) to tti, mower 
owners, and those (xi,x 2 ) values which fall into R 2 to 77 - 2 , nonowners, we would 
make some mistakes. Some riding-mower owners would be incorrectly classified as 
nonowners and, conversely, some nonowners as owners.The idea is to create a rule 
(regions Ri and R 2 ) that minimizes the chances of making these mistakes. (See 
Exercise 11.2.) ■ 

A good classification procedure should result in few misclassifications. In other 
words, the chances, or probabilities, of misclassification should be small. As we shall 
see, there are additional features that an “optimal” classification rule should possess. 

It may be that one class or population has a greater likelihood of occurrence 
than another because one of the two populations is relatively much larger than the 
other. For example, there tend to be more financially sound firms than bankrupt 
firms. As another example, one species of chickweed may be more prevalent than 
another. An optimal classification rule should take these “prior probabilities of 
occurrence” into account. If we really believe that the (prior) probability of a finan¬ 
cially distressed and ultimately bankrupted firm is very small, then one should 


Lot size in thousands of square feet 
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X] Figure I l.l Income and lot size 
for riding-mower owners and 
nonowners. 


classify a randomly selected firm as nonbankrupt unless the data overwhelmingly 
favors bankruptcy. 

Another aspect of classification is cost. Suppose that classifying a 77-1 object as 
belonging to 77- 2 represents a more serious error than classifying a 7 t - 2 object as be¬ 
longing to 77 "j. Then one should be cautious about making the former assignment. As 
an example, failing to diagnose a potentially fatal illness is substantially more “cost¬ 
ly” than concluding that the disease is present when, in fact, it is not. An optimal 
classification procedure should, whenever possible, account for the costs associated 
with misclassification. 

Let /i(x) and / 2 (x) be the probability density functions associated with the 
P X 1 vector random variable X for the populations 77 ^ and 7 r 2 , respectively. An ob¬ 
ject with associated measurements x must be assigned to either 77 ^ or tt- 2 . Let fl be 
the sample space—that is, the collection of all possible observations x. Let R\ be that 
set of x values for which we classify objects as tt 1 and R 2 = fl-R 1 be the remaining 
x values for which we classify objects as 7 r 2 . Since every object must be assigned to 
one and only one of the two populations, the sets Ri and R 2 are mutually exclusive 
and exhaustive. For p = 2, we might have a case like the one pictured in Figure 11.2. 

The conditional probability, P( 211), of classifying an object as 77- 2 when, in fact, 
it is from 77 "x is 

P(2I1) = /’(Xe^lTT-j) = f /i(x) dx (11-1) 

Similarly, the conditional probability, F(112), of classifying an object as 7 t 1 when it 
is really from 7 t 2 is 

F( 1 I 2 ) = F(Xe* 1 l 77 2 ) = f f 2 (x) dx 
J Ry 



( 11 - 2 ) 
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Figure 11.2 Classification regions 
for two populations. 


The integral sign in (11-1) represents the volume formed by the density function 
f,(x) over the region R 2 . Similarly, the integral sign in (11-2) represents the volume 
formed by / 2 (x) over the region ft,.This is illustrated in Figure 11.3 for the univari¬ 
ate case, p = 1. . 

Let Pl be the prior probability of i r, and p 2 be the prior probability of tt 2 , 
where A + p 2 = 1. Then the overall probabilities of correctly or incorrectly clas¬ 
sifying objects can be derived as the product of the prior and conditional classifi- 
cation probabilities: 

^(observation is correctly classified as tti) = P (observation comes from ni 

and is correctly classified as 7 rj) 

= P(Xe i?il7r 1 )/ > (7r 1 ) = P( 11 l)pi 

P( observation is misclassified as t^) = ^observation comes from tt 2 

and is misclassified as 7r t ) 

= P(\ 6 R\ 1 7T2)P( 7T 2) = P i^ 2 )Pl 

P(observation is correctly classified as tt 2 ) = ^(observation comes from tt 2 

and is correctly classified as 7r 2 ) 

- P(XEi? 2 l7r 2 )P(7r 2 ) = P{2 I 2)pi 


P(2\\) = jfi (x)dx 



Figure 11.3 Misclassification probabilities for hypothetical classification regions 
when p = 1. 
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P {observation is misclassified as 7r 2 ) = P(observation comes from tt\ 

and is misclassified as 7r 2 ) 

= P{X^R 1 \ir l )P{ir x ) » P( 2 \\)p x 



| Classify as: 


TTi 7T 2 

^1 

0 c(2ll) 

7T 2 

c(H2) 0 


Classification schemes are often evaluated in terms of their misclassification 
probabilities (see Section 11.4), but this ignores misclassification cost. For example, 
even a seemingly small probability such as .06 = P(2 11) may be too large if the cost 
of making an incorrect assignment to 7 t 2 is extremely high . A rule that ignores costs 
may cause problems. 

The costs of misclassification can be defined by a cost matrix: 


True population: 


The costs are (1) zero for correct classification, (2) c(l 12) when an observation from 
7 r 2 is incorrectly classified as 7 and ( 3 ) c (2 II) when a 77 -j observation is incorrect¬ 
ly classified as 7r 2 . 

For any rule, the average, or expected cost of misclassification (ECM) is provid¬ 
ed by multiplying the off-diagonal entries in (11-4) by their probabilities of occur¬ 
rence, obtained from (11-3). Consequently, 

ECM = c(2ll)P(2ll)p 1 + c(1I2)P(1I2)/? 2 (11-5) 

A reasonable classification rule should have an ECM as small, or nearly as 
small, as possible. 

Result I l.l. The regions and R 2 that minimize the ECM are defined by the 
values x for which the following inequalities hold: 

/i(*) „ / c (i 12) ^ (pA 
'■ / 2 (x) U(2I1)J \Pi/ 


density 

ratio 


prior 

probability 

ratio 


c(H2) 

c(2ll) 


density 

ratio 


prior 

probability 

ratio 


Proof. See Exercise 11.3. 


It is clear from (11-6) that the implementation of the minimum ECM rule re¬ 
quires (1) the density function ratio evaluated at a new observation x 0 , (2) the cost 
ratio, and (3) the prior probability ratio.The appearance of ratios in the definition of 
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the optimal classification regions is significant. Often, it is much easier to specify the 
ratios than their component parts. 

For example, it may be difficult to specify the costs (in appropriate units) of 
classifying a student as college material when, in fact, he or she is not and classifying 
a student as not college material, when, in fact, he or she is. The cost to taxpayers of 
educating a college dropout for 2 years, for instance, can be roughly assessed. The 
cost to the university and society of not educating a capable student is more difficult 
to determine. However, it may be that a realistic number for the ratio of these mis- 
classification costs can be obtained. Whatever the units of measurement, not admit¬ 
ting a prospective college graduate may be five times more costly, over a suitable 
time horizon, than admitting an eventual dropout. In this case, the cost ratio is five. 

It is interesting to consider the classification regions defined in (11-6) for some 
special cases. 


Special Cases of Minimum Expected Cost Regions 


(a) P 2 /P 1 = 1 (equal prior probabilities) 

/i( x ) ^ c{ 112) AW 

M*) “C(211) * 2 ‘ A(x) 

(b) c(l!2)/c(2ll) = 1 (equal misclassificationcosts) 


/■(*) c(l\2) 

m c( 211) 


fi( x ) ^ Pi 
fi(*) ~ Pi 


AW c Pi 
AW Pi 


(C) P 2 /P 1 = C(ll2)/c(2ll) = lor P 2 /PI = l/(c(112)/c(211)) 
(equal prior probabilities and equal misclassification costs) 


When the prior probabilities are unknown, they are often taken to be equal, and 
the minimum ECM rule involves comparing the ratio of the population densities to 
the ratio of the appropriate misclassification costs. If the misclassification cost ratio 
is indeterminate, it is usually taken to be unity, and the population density ratio is 
compared with the ratio of the prior probabilities. (Note that the prior probabilities 
are in the reverse order of the densities.) Finally, when both the prior probabili¬ 
ty and misclassification cost ratios are unity, or one ratio is the reciprocal of the 
other, the optimal classification regions are determined simply by comparing the 
values of the density functions. In this case, if Xo is a new observation and 
/i(xo)// 2 ( x o) — 1 —that is,fi(x 0 ) > / 2 (x 0 ) —we assign x 0 to 7 ^. On the other hand, 
if fi( x o)/fi( x o) < 1, or /i(x 0 ) < / 2 (x 0 ), we assign x 0 to tt 2 . 

It is common practice to arbitrarily use case (c) in (11-7) for classification. This 
is tantamount to assuming equal prior probabilities and equal misclassification costs 
for the minimum ECM rule. 2 

2 This is the justification generally provided. It is also equivalent to assuming the prior probability 
ratio to be the reciprocal of the misclassification cost ratio. 
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Example 11.2 (Classifying a new observation into one of the two populations) A re¬ 
searcher has enough data available to estimate the density functions / x (x) and / 2 (x) 
associated with populations 7 and 7r 2 , respectively. Suppose c(211) = 5 units and 
c(l 12) = 10 units. In addition, it is known that about 20% of all objects (for which' 
the measurements x can be recorded) belong to 7 r 2 . Thus, the prior probabilities are 
Pi = .8 and fh = .2. 

Given the prior probabilities and costs of misclassification, we can use (11-6) to 
derive the classification regions Ri and R 2 . Specifically, we have 



Suppose the density functions evaluated at a new observation x 0 give /i(x 0 ) = -3 
and / 2 (x 0 ) = .4. Do we classify the new observation as tt 1 or 7 t 2 ? To answer the 
question, we form the ratio 

/i(*o) _ .3 _ 

/ 2 (*0) -4 ' 

and compare it with .5 obtained before. Since 


/i(*o) = 7S / c( 112) \ fp{\ 

/ 2 (x 0 ) ' \c(2n)J \pj 


.5 


we find that x 0 e R j and classify it as belonging to tti . 


Criteria other than the expected cost of misclassification can be used to 
derive “optimal” classification procedures. For example, one might ignore the costs 
of misclassification and choose Rj and R 2 to minimize the total probability of 
misclassification (TPM): 

TPM = P(misclassifying a 77-1 observation or misclassifying a 7 r 2 observation) 

= P(observation comes from n 1 and is misclassified) 

-I- P (observation comes from 7 t 2 and is misclassified) 

= Pi f /i(x)rf* + P2 f f2(x)dx (11-8) 

JR 2 JR 1 

Mathematically, this problem is equivalent to minimizing the expected cost of 
misclassification when the costs of misclassification are equal. Consequently, the 
optimal regions in this case are given by (b) in (11-7). 
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We could also allocate a new observation x 0 to the population with the largest 
“posterior” probability P^Ixq). By Bayes’s rule, the posterior probabilities are 


/ > (t7iIx 0 ) 


P{tt 1 occurs and we observe x 0 ) 

P( we observe x 0 ) 

P( we observe x 0 l 77 1 )/ > (7r 1 ) 

P(we observe Xq 177 1 )/ > (77 ]) + P (we observe x 0 1 772 )^( 77 ^ 
Pi/i(*o) 

Pl/l(*o) + 

, Pifrixo) 


P(tt 2 \x 0 ) = 1 - P(T7j lx 0 ) = 


Pl/l(*o) + Plf 2 (*o) 


Classifying an observation x 0 as 77, when P{it\ I x 0 ) > P(tt 2 I x 0 ) is equivalent to 
using the (b) rule for total probability of misclassification in (11-7) because the de¬ 
nominators in (11-9) are the same. However, computing the probabilities of the pop¬ 
ulations 77! and 77 2 after observing x 0 (hence the name posterior probabilities) is 
frequently useful for purposes of identifying the less clear-cut assignments. 


11.3 Classification with Two Multivariate Normal Populations 

Classification procedures based on normal populations predominate in statistical 
practice because of their simplicity and reasonably high efficiency across a wide va¬ 
riety of population models. We now assume that /i(x) and / 2 (x) are multivariate 
normal densities, the first with mean vector /ij and covariance matrix 2 1 and the 
second with mean vector /i 2 and covariance matrix X 2 * 

The special case of equal covariance matrices leads to a particularly simple lin¬ 
ear classification statistic. 


Classification of Normal Populations When X| = X 2 = 2 

Suppose that the joint densities of X' = [X\,X 2 . X p ] for populations tt 1 and tt 2 

are given by 


/,(*) = 


(27r)P/ 2 |S| l/2< 


exp --(x - m/)'2 \x - n;) for/ = 1,2 (11-10) 


Suppose also that the population parameters /ii,/i 2 , and 2 are known. Then, after 
cancellation of the terms (2tt) p ^\ 5 | 1/2 the minimum ECM regions in (11-6) become 

Rp- exp -|(x - “ Mi) + |(* - M 2 )'S-‘(x - M 2 ) 

" > (sm\ (ei) 

\c(2ll)/ W 

Ri- exp -i(x - Mi)'2 _1 (x - Ml) + ^(x - M2)'2 _, (x - M 2 ) 


c(ll2)\ (p£ 

c(211) / \ Pl/ 


(11-11) 
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Given these regions R x and R 2 , we can construct the classification rule given in the 
following result. 

Result 11.2. Let the populations 77! and rr 2 be described by multivariate normal 
densities of the form (11-10).Then the allocation rule that minimizes the ECM is as 
follows: 

Allocate x 0 to 77! if 

(Mi - M2)'2" 1 x 0 - ^( Ml - M2)'2 TVi + M 2 ) a In (U-12) 

Allocate x 0 to 77 2 otherwise. 

Proof. Since the quantities in (11-11) are nonnegative for all x, we can take their 
natural logarithms and preserve the order of the inequalities. Moreover (see 
Exercise 11.5), 

~\( x _ Mi)’2 _1 (x - Ml) + |(x - M 2 )'2 _1 (x - M 2 ) 

= (Mi “ M2)'2-*x - |(Mi ~ M 2 )'2" 1 (Mi + M 2 ) (U-13) 

and, consequently, 

R i- (Mi ~ M2)'2 T 1 x - |(mi - M2)'2T 1 (Mi + M 2 ) s In j 

Ri- (Mi - M2)'2 _1 x-i(Mi - M2)'2 _1 (Mi + M 2 ) < In ^ll) ) (^) 

(11-14) 

The minimum ECM classification rule follows. ■ 

In most practical situations, the population quantities Mi, /^ 2 » and X are un¬ 
known, so the rule (11-12) must be modified. Wald [31] and Anderson [2] have sug¬ 
gested replacing the population parameters by their sample counterparts. 

Suppose, then, that we have n l observations of the multivariate random vari¬ 
able X' = [X\ , X 2 ,..., X^ from 77 ! and n 2 measurements of this quantity from 7 r 2 , 
with rii + n 2 — 2 ^ p. Then the respective data matrices are 



(11-15) 
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From these data matrices, the sample mean vectors and covariance matrices are 
determined by 

*1 s i =— 

0?xl) ni y = i (pxp) Tl\ 1 >=1 /-II 

1 ZJ2 y n 2 ' ' 

*2 = —E* 2 ;, S 2 --r2(*2;- 5 2)(*2, - *2)' 

(/?xi) n 2 y=1 ( pX p) n 2 - 1 y —1 

Since it is assumed that the parent populations have the same covariance matrix X, 
the sample covariance matrices Si and S 2 are combined (pooled) to derive a single, 
unbiased estimate of X as in (6-21). In particular, the weighted average 

Spoo,ed = i)_ Sl + _K-i) 2 +(n 2 -i)_ Sz (11 ' 17) 

is an unbiased estimate of X if the data matrices X! and X 2 contain random sam¬ 
ples from the populations 77-j and tt 2 , respectively. 

Substituting x x for fi x , x 2 for /* 2 , and S poo ied f° r ^ m (11-12) gives the “sample” 
classification rule: 


The Estimated Minimum ECM Rule for Two Normal Populations 

Allocate x 0 to tt\ if 

1 _ - [Y‘(1'2)Vp 2 V 

(*1 - X 2 )'Sp^ olcd X 0 - -(Xl - X 2 )'Spio]ed(Xl + * 2 ) a ln |^ C ( 2 H) j [yj 

(11-18) 


Allocate x 0 to n 2 otherwise. 


c(U2) \ /pA = 2 


If, in (11-18), 


Vc(2ii); \pij 

then ln(l) = 0, and the estimated minimum ECM rule for two normal populations 
amounts to comparing the scalar variable 

y = ( X 1 - X 2 )'S^oled x = »'* (11-19) 

evaluated at x 0 , with the number 


rh = ~(X! - X 2 )'Spi„led(*l + * 2 ) 

= j(yi + h) 


( 11 - 20 ) 


yi = (Xj - Xzj'Spijied*! = a '*i 


and 


ft = (*1 - *2)'Spi>led*2 = *'*2 
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That is, the estimated minimum ECM rule for two normal populations is tanta¬ 
mount to creating two univariate populations for the y values by taking an appropri¬ 
ate linear combination of the observations from populations 7 rj and 7r 2 and then 
assigning a new observation x 0 to 7 r 1 or 7 t 2 , depending upon whether % ~ a'x 0 falls 
to the right or left of the midpoint rh between the two univariate means y x and y 2 . 

Once parameter estimates are inserted for the corresponding unknown popula¬ 
tion quantities, there is no assurance that the resulting rule will minimize the ex¬ 
pected cost of misclassification in a particular application. This is because the 
optimal rule in (11-12) was derived assuming that the multivariate normal densities 
/j(x) and / 2 (x) were known completely. Expression (11-18) is simply an estimate of 
the optimal rule. However, it seems reasonable to expect that it should perform well 
if the sample sizes are large. 3 

To summarize, if the data appear to be multivariate normal 4 , the classification 
statistic to the left of the inequality in (11-18) can be calculated for each new obser¬ 
vation x 0 . These observations are classified by comparing the values of the statistic 
with the value of ln[(c(l 12)/c(211 ))(p 2 /p\)]. 


Example 11.3 (Classification with two normal populations—common X and equal 
costs) This example is adapted from a study [4] concerned with the detection of 
hemophilia A carriers. (See aJso Exercise 11.32.) 

To construct a procedure for detecting potential hemophilia A carriers, blood 
samples were assayed for two groups of women and measurements on the two 
variables, 

X x ~ log 10 (AHF activity) 

X 2 = log 10 (AHF-like antigen) 

recorded. (“AHF” denotes antihemophilic factor.) The first group of n x ~ 30 
women were selected from a population of women who did not carry the hemophilia 
gene. This group was called the normal group. The second group of n 2 = 22 women 
was selected from known hemophilia A carriers (daughters of hemophiliacs, 
mothers with more than one hemophilic son, and mothers with one hemophilic son 
and other hemophilic relatives). This group was called the obligatory carriers. The 
pairs of observations (x ]y x 2 ) for the two groups are plotted in Figure 11.4. Also 
shown are estimated contours containing 50% and 95% of the probability for 
bivariate normal distributions centered at x 3 and x 2 , respectively. Their common 
covariance matrix was taken as the pooled sample covariance matrix S poo | e d- I n Ibis 
example, bivariate normal distributions seem to fit the data fairly well. 

The investigators (see [4]) provide the information 



3 As ihe sample sizes increase, x ] ,x 2 , and Sp^d become, with probability approaching 1,indistin¬ 
guishable from fi \, /i 2 , and X, respectively [see (4-26) and (4-27)]. 

4 At the very least, the marginal frequency distributions of the observations on each variable can be 
checked for normality. This must be done for the samples from both populations. Often, some variables 
must be transformed in order to make them more “normal looking.” (See Sections 4.6 and 4.8.) 
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Figure 11.4 Scatter plots of [Jog 10 (AHF activity), logi 0 (AHF-like antigen)] for the 
normal group and obligatory hemophilia A carriers. 


131.158 - 90.423 
-90.423 108.147 


Therefore, the equal costs and equ^l priors discriminant function [see (11-19)] is 


y = a x = [Xj 


*2] Spooled J 


.2418 -.0652] 


131.158 -90.423 x x 
-90.423 108.147 x 2 


= 37.61xi 


Moreover, 


yi = a'xj = [37.61 -28.92] 


K = a'x 2 = [37.61 -28.92] 


= - 10.10 


and the midpoint between these means [see (11-20)] is 

m = \{y\ + yz) = | (.88 - 10.10) = -4.61 

Measurements of AHF activity and AHF-like antigen on a woman who may be 
a hemophilia A carrier give X\ = —.210 and x 2 — —.044. Should this woman be clas 
sified as tt 1 (normal) or ir 2 (obligatory carrier)? 

Using (11-18) with equal costs and equal priors so that ln(l) = 0, we obtain 

Allocate x 0 to 77 ^ if = a'x 0 > m = -4.61 
Allocate x 0 to tt 2 if y Q = a'x 0 < in = -4.61 
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where x' 0 = [-.210, -.044]. Since 

y 0 = a'x 0 = [37.61 -28.92] 


= -6.62 < 


we classify the woman as 7 t 2 , an obligatory carrier. The new observation is indicated 
by a star in Figure 11.4. We see that it falls within the estimated .50 probability con¬ 
tour of population 7 t 2 and about on the estimated .95 probability contour of popula¬ 
tion 77 j. Thus, the classification is not clear cut. 

Suppose now that the prior probabilities of group membership are known. For 
example, suppose the blood yielding the foregoing X\ and x 2 measurements is drawn 
from the maternal first cousin of a hemophiliac. Then the genetic chance of being a 
hemophilia A carrier in this case is .25. Consequently, the prior probabilities of 
group membership are p x = .75 and = .25. Assuming, somewhat unrealistically, 
that the costs of misclassification are equal, so that c(l 12) = c(211), and using the 
classification statistic 


&=(*i- X 2 )'S p 


K*1 - * 2 )*Spooled (x, + X 2 ) 


or w = a'x 0 - m with x' 0 = [-.210, -.044], m = -4.61, and a'x 0 
have 

w = -6.62 - (-4.61) = -2.01 
Applying (11-18), we see that 


-6.62, we 


w = -2.01 < In 


2] 1 P 2 

rj ,n b 


and we classify the woman as 7 t 2 , an obligatory carrier. ■ 

Scaling 

The coefficient vector a = Sp^j^ (xi - x 2 ) is unique only up to a multiplicative 
constant, so, for c ^ 0, any vector ca will also serve as discriminant coefficients. 

The vector a is frequently “scaled” or “normalized” to ease the interpretation of 
its elements. Two of the most commonly employed normalizations are 


so that a* has unit length. 
2. Set 


a* =4 
a 1 


( 11 - 21 ) 


( 11 - 22 ) 


so that the first element of the new coefficient vector a* is 1. 


In both cases, a* is of the form ca. For normalization (1), c = (a'a) ^ and 
for (2), c = a[ l . 
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The magnitudes of a*, a 2 ,...,a* p in (11-21) all lie in the interval [-1,1]. i n 
(11-22), a* = 1 and a 2 ,...,a* p are expressed as multiples of a *. Constraining the a* 
to the interval [-1,1] usually facilitates a visual comparison of the coefficients. Sim¬ 
ilarly, expressing the coefficients as multiples of allows one to readily assess the 
relative importance (vis-a-vis X\) of variables X 2 ,... ,X p as discriminators. 

Normalizing the a {s is recommended only if the X variables have been stan¬ 
dardized. If this is not the case, a great deal of care must be exercised in interpreting 
the results. 


Fisher’s Approach to Classification with Two Populations 

Fisher [10] actually arrived at the linear classification statistic (11-19) using an en¬ 
tirely different argument. Fisher’s idea was to transform the multivariate observa¬ 
tions x to univariate observations y such that the y’s derived from population tt x and 
n 2 were separated as much as possible. Fisher suggested taking linear combinations 
of x to create y’s because they are simple enough functions of the x to be handled 
easily. Fisher’s approach does not assume that the populations are normal. It does, 
however, implicitly assume that the population covariance matrices are equal, be¬ 
cause a pooled estimate of the common covariance matrix is used. 

A fixed linear combination of the x’s takes the values y n , y l2 ,..., y ] „ x for the 
observations from the first population and the values ft { , >* 22 ,.. ■, ftn 2 f° r the obser¬ 
vations from the second population. The separation of these two sets of univariate 
y’s is assessed in terms of the difference between ft and ft. expressed in standard 
deviation units. That is, 

11 2O'!/ - h) 2 + 2te; - yz) 2 

I y\ yi 1 2 m /=] 

separation =-. where s y =--- 

Sy w 2 ”1- n 2 2 

is the pooled estimate of the variance. The objective is to select the linear combina¬ 
tion of the x to achieve maximum separation of the sample means ft and 


Result 11.3. The linear combination y = a'x = (x 2 - X 2 ) Spi 0 ied x maximizes the 
ratio 

/ squared distance \ 

^between sample means of y) {ft ~ ft) 1 

(sample variance of y) s 2 y 

= (a f xj - a'x 2 ) 2 
a Spooled 3 


(i'Af 

^ Spools a 


(11-23) 


over all possible coefficient vectors a where d = (x 2 - x 2 ). The maximum of the 
ratio (11-23) is D 2 = (x 2 - x 2 ) , S^ o}ed (x ] - x 2 ). 
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Proof. The maximum of the ratio in (11-23) is given by applying (2-50) directly. 
Thus, setting d = (x 2 - x 2 ), we have 


max - 
fi a Sry 


= d'S^iedd = (Xj - x 2 )'S p i,| ed (x ] - x 2 ) = D 2 


where D 2 is the sample squared distance between the two means. 
Note that sj in (11-33) may be calculated as 

2 (yv - yi) 2 + 2 (*; - hf 

„2 _ jzi _>=>_ 


n i + n 2 - 2 


(11-24) 


with y 1} = a'x ly and ft, = a'x 2y . 


Example 11.4 (Fisher's linear discriminant for the hemophilia data) Consider the 
detection of hemophilia A carriers introduced in Example 11.3. Recall that the equal 
costs and equal priors linear discriminant function was 

y - a'x = (x 2 - x 2 )'Spo 0]ed x = 37.61 jc 2 - 28.92jc 2 

This linear discriminant function is Fisher’s linear function, which maximally 
separates the two populations, and the maximum separation in the samples is 

D 2 = (I, - x^'S^^x, - x 2 ) 

= [.2418, -.0652] 13L158 _90 - 423 2418 

J [_-90.423 108.147J ,0652_ 

= 10.98 ■ 

Fisher’s solution to the separation problem can also be used to classify new 
observations. 


An Allocation Rule Based on Fisher’s Discriminant Function 5 

Allocate x 0 to i t 1 if 

yo ~ ( X 1 — x 2) spooled x 0 

a m = |(x, - x 2 )'S^ oled (x] + x 2 ) 


Allocate x 0 to — 2 if 


A) - m a 0 
Po < m 

yo~ m < 0 


(11-25) 


5 We must have («! + n 2 ~ 2) > p; otherwise Sp^d is singular, and the usual inverse, Sp^jed. does 
not exist. 
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* 


Figure I L5 A pictorial representation of Fisher’s procedure for two populations 
with p = 2. 

The procedure (11-23) is illustrated, schematically, for p = 2 in Figure 11.5. All 
points in the scatter plots are projected onto a line in the direction a, and this direc¬ 
tion is varied until the samples are maximally separated. 

Fisher’s linear discriminant function in (11-25) was developed under the as¬ 
sumption that the two populations, whatever their form, have a common covariance 
matrix. Consequently, it may not be surprising that Fisher’s method corresponds to 
a particular case of the minimum expected-cost-of-misclassification rule. The first 
term, y = (*i - in the classification rule (11-18) is the linear function 

obtained by Fisher that maximizes the univariate “between” samples variability rel¬ 
ative to the “within” samples variability. [See (11-23).] The entire expression 

w = (xj - )'Spooled* ~ |(*i ~ *2)'S p ioled (*1 + * 2 ) 

= (*1 - * 2 ) 'Spooled [x - |(Xl + * 2 )] (11-26) 

is frequently called Anderson's classification function (statistic). Once again, if 
[(c(ll2)/c(2!l))(p2/Pi)] = 1> so that ln[(c(l l2)/c(21 l))(p 2 /pi)] = 0, Rule 
(11-18) is comparable to Rule (11-26), based on Fisher’s linear discriminant func¬ 
tion. Thus, provided that the two normal populations have the same covariance ma¬ 
trix, Fisher’s classification rule is equivalent to the minimum ECM rule with equal 
prior probabilities and equal costs of misclassification. 

Is Classification a Good Idea? 

For two populations, the maximum relative separation that can be obtained by 
considering linear combinations of the multivariate observations is equal to the 
distance D 2 . This is convenient because D 2 can be used, in certain situations, to test 
whether the population means jii and /* 2 differ significantly. Consequently, a test 
for differences in mean vectors can be viewed as a test for the “significance” of the 
separation that can be achieved. 
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Suppose the populations rr l and 7r 2 are multivariate normal with a common co- 
variance matrix X. Then, as in Section 6.3, a test of Hq\ /aj = fi 2 versus H\\ ^ /* 2 

is accomplished by referring 

/ + n 2 - P - 1 \ ( n iw 2 \ , 

V ("l + "2 - 2 )p ) + n 2 ) 

to an /-distribution with = p and v 2 = + n 2 — p — 1 d.L If Ho is rejected, 

we can conclude that the separation between the two populations 7ii and 7r 2 is 
significant. 

Comment. Significant separation does not necessarily imply good classifica¬ 
tion. As we shall see in Section 11.4, the efficacy of a classification procedure can be 
evaluated independently of any test of separation. By contrast, if the separation is 
not significant, the search for a useful classification rule will probably prove 
fruitless. 


Classification of Normal Populations When 2| # X 2 

As might be expected, the classification rules are more complicated when the popu¬ 
lation covariance matrices are unequal. 

Consider the multivariate normal densities in (11-10) with X,-, i — 1,2, replac¬ 
ing X. Thus, the covariance matrices, as well as the mean vectors, are different from 
one another for the two populations. As we have seen, the regions of minimum 
ECM and minimum total probability of misclassification (TPM) depend on the 
ratio of the densities, fi(x)/f 2 (\) > or, equivalently, the natural logarithm of the den¬ 
sity ratio, ln[/i(x)// 2 (x)] = ln[/i(x)] - ln[/ 2 (x)]. When the multivariate normal 
densities have different covariance structures, the terms in the density ratio involv¬ 
ing | X, | 1/2 do not cancel as they do when Xi = X 2 . Moreover, the quadratic forms in 
the exponents of /i(x) and / 2 (x) do not combine to give the rather simple result in 
(11-13). 

Substituting multivariate normal densities with different covariance matrices 
into (11-6) gives, after taking natural logarithms and simplifying (see Exercise 
11.15), the classification regions 

R ' : 4 x ' (Zr '- 2J1)x + ( ^ 2r ‘“ ^ 1)x _(§iiy)(f)_ 


R 2 : -ix'(Sr 1 -S 2 - 1 )x+(MiSr 1 -M2^ 1 )x-A:<ln 

where 

k = f ln ({f^j) + |(/*12lVi - 



(11-27) 

(11-28) 


The classification regions are defined by quadratic 4 functions of x. When Xi = X 2 , 
the quadratic term, — Ix'fXj 1 — XJ 1 )*, disappears, and the regions defined by 
(11-27) reduce to those defined by (11-14). 
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The classification rule for general multivariate normal populations follows 
directly from (11-27). 

Result 11.4. Let the populations and 7 t 2 be described by multivariate normal 
densities with mean vectors and covariance matrices ^i,X 1 and M 2 >^ 2 > respec¬ 
tively. The allocation rule that minimizes the expected cost of misclassification is 
given by 

Allocate x 0 to 7t x if 


•"Z&csr 1 - ^ 2 _1 )xo + (MiXr 1 - M 2 ^)X 0 - * a In 
Allocate x 0 to 7 r 2 otherwise. 

Here k is set out in (11-28). 


( c(H2)^ (Pi 


\d 211)7 \PiJ J 


In practice, the classification rule in Result 11.5 is implemented by substituting 
the sample quantities xi, x 2 , S 1? and S 2 (see (11-16)) for Mi* M 2 * ^ 1 * an d S 2 , 
respectively. 6 


Quadratic Classification Rule 

(Normal Populations with Unequal Covariance Matrices) 


Allocate x 0 to 7 rj if 

-ix^fsr 1 - S 2 1 ) x 0 + (iisr 1 - x^SJ^Xo - k S In 


Allocate x 0 to 7 t 2 otherwise. 


/ c(H2) \ (pf 
(c(2ll)/ { Pl )j 

(11-29) 


Classification with quadratic functions is rather awkward in more than two di¬ 
mensions and can lead to some strange results. This is particularly true when the 
data are not (essentially) multivariate normal. 

Figure 11.6(a) shows the equal costs and equal priors rule based on the ideal¬ 
ized case of two normal distributions with different variances. This quadratic rule 
leads to a region Ri consisting of two disjoint sets of points. 

In many applications, the lower tail for the 7 ^ distribution will be smaller 
than that prescribed by a normal distribution. Then, as shown in Figure 11.6(b), 
the lower part of the region R u produced by the quadratic procedure, does not 
line up well with the population distributions and can lead to large error rates. 
A serious weakness of the quadratic rule is that it is sensitive to departures from 
normality. 

6 The inequalities n x > p and n 2 > p must both hold for Sf l and SJ 1 to exist. These quantities are 
used in place of IJ 1 and X 2 1 > respectively, in the sample analog (11-29). 






(a) 



Figure 11.6 Quadratic rules for (a) two normal distribution with unequal variances 
and (b) two distributions, one of which is nonnormal—rule not appropriate. 


If the data are not multivariate normal, two options are available. First, the non- 
normal data can be transformed to data more nearly normal, and a test for the 
equality of covariance matrices can be conducted (see Section 6.6) to see whether 
the linear rule (11-18) or the quadratic rule (11-29) is appropriate. Transformations 
are discussed in Chapter 4. (The usual tests for covariance homogeneity are greatly 
affected by nonnormality. The conversion of nonnormal data to normal data must 
be done before this testing is carried out.) 

Second, we can use a linear (or quadratic) rule without worrying about the form 
of the parent populations and hope that it will work reasonably well. Studies (see 
[22] and [23]) have shown, however, that there are nonnormal cases where a linear 
classification function performs poorly, even though the population covariance ma¬ 
trices are the same. The moral is to always check the performance of any classifica¬ 
tion procedure. At the very least, this should be done with the data sets used to build 
the classifier. Ideally, there will be enough data available to provide for “training” 
samples and “validation” samples. The training samples can be used to develop 
the classification function, and the validation samples can be used to evaluate its 
performance. 





596 Chapter 11 Discrimination and Classification 


11.4 Evaluating Classification Functions 

One important way of judging the performance of any classification procedure is to 
calculate its “error rates,” or misclassification probabilities. When the forms of the 
parent populations are known completely, misclassification probabilities can be cab 
culated with relative ease, as we show in Example 11.5. Because parent populations 
are rarely known, we shall concentrate on the error rates associated with the sample 
classification function. Once this classification function is constructed, a measure of 
its performance in future samples is of interest. 

From (11-8), the total probability of misclassification is 

TPM = Pi f /i(x) dx + p 2 f f 2 (x) dx 
Jr 2 J R \ 

The smallest value of this quantity, obtained by a judicious choice of R x and R 2 , is 
called the optimum error rate (OER). 

Optimum error rate (OER) = pi / /i(x) dx + ^ / / 2 (x) dx (11-30) 

Jr 2 JR\ 

where R\ and R 2 are determined by case (b) in (11-7). 

Thus, the OER is the error rate for the minimum TPM classification rule. 


Example 11.5 (Calculating misclassification probabilities) Let us derive an expres¬ 
sion for the optimum error rate when P\ = p 2 = \ and /i(x) and / 2 (x) are the mul¬ 
tivariate normal densities in (11-10). 

Now, the minimum ECM and minimum TPM classification rules coincide when 
c(l 12) = c(211). Because the prior probabilities are also equal, the minimum 
TPM classification regions are defined for normal populations by (11-12), with 


c(V2) 
C( 211) 


= 0. We find that 


^l : (Ml P'lY'Z lx 2(^1 M 2 ) ^ *(Mi + M 2 ) 


^ 2 - (Mi - M 2 )'X ’x - |(mi - M2)'2 *(Mi + M 2 ) < 0 
These sets can be expressed in terms of y = (p 1 - /* 2 )'£ -1 x = a'xas 

y - 2(mi - M2)'X->(mi + M2) 

^2 ( y )- y < |(mi - m2)'£‘~ 1 (mi + M2) 

But Y is a linear combination of normal random variables, so the probability densi 
ties of y, f\(y) and f 2 (y ), are univariate normal (see Result 4.2) with means and a 
variance given by 

Miy = a'Mi = (Ml ~ M2)'2T 1 Mi 
M 2 y ~ a>2 = (Mi - M 2 )'2- ] M2 
a y = a'Xa = (m - M2)'£ _1 (mi ~ M 2 ) = A 2 
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Now, 

TPM = \ P[misclassifying a ir Y observation as 7 t 2 ] 

+ \Pfmisclassifying a 7 t 2 observation as tti] 

But, as shown in Figure 11.7 

P[misclassifying a 7T! observation as tt 2 ] = P(2I1) 

= P[Y < — M2)'X -1 (/*! + M2)] 

y - Ml Y < 2(^1 ~ M 2 ) / X“ 1 (Mi + M2) - (Ml ~~ M2) , X~ 1 M i 
o’y A 




where <t> (*) is the cumulative distribution function of a standard normal random 
variable. Similarly, 

Pjmisclassifying a 7 t 2 observation as tt ] ] 

= P{ 112) = P[Y a |( M , - /t 2 )'2- 1 ( M i + M 2 )] 



Therefore, the optimum error rate is 

OER = minimum TPM = ^ = (11-31) 

If, for example, A 2 = ( Ml - mJ'X'Vi - M 2 ) = 2.56, then A = V236 = 1.6, and, 
using Table 1 in the appendix, we obtain 


Minimum TPM = <J> 



= <f>(—.8) = .2119 


The optimal classification rule here will incorrectly allocate about 21% of the items 
to one population or the other. ■ 


Example 11.5 illustrates how the optimum error rate can be calculated when the 
population density functions are known. If as is usually the case, certain population 
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parameters appearing in allocation rules must be estimated from the sample, then 
the evaluation of error rates is not straightforward. 

The performance of sample classification functions can, in principle, be evaluat¬ 
ed by calculating the actual error rate (AER), 

AER = pi I /i(x) dx + p 2 f f 2 (x)dx (11-32) 

JRj JRy 

where R x and R 2 represent the classification regions determined by samples of size 
rii and n 2 , respectively. For example, if the classification function in (11-18) is 
employed, the regions and R 2 are defined by the set of x’s for which the following 
inequalities are satisfied. 


R^. (xj - x 2 ) , S p i 0 | ed x 

■^2* (*1 ~ x 2) Sp 00 i et j x 



The AER indicates how the sample classification function will perform in future 
samples. Like the optimal error rate, it cannot, in general, be calculated, because it 
depends on the unknown density functions /j(x) and / 2 (x). However, an estimate of 
a quantity related to the actual error rate can be calculated, and this estimate will be 
discussed shortly. 

There is a measure of performance that does not depend on the form of the 
parent populations and that can be calculated for any classification procedure. This 
measure, called the apparent error rate (APER), is defined as the fraction of observa¬ 
tions in the training sample that are misclassified by the sample classification function. 

The apparent error rate can be easily calculated from the confusion matrix , 
which shows actual versus predicted group membership. For n x observations from 
77 -j and n 2 observations from 7 r 2 , the confusion matrix has the form 


Actual 77 1 

membership 


Predicted membership 
77 1 77 2 



(11-33) 


where 

n ic — number of 7t 1 items correctly classified as 77-j items 
n lM - number of 77 -j items misclassified as 7r 2 items 
n 2C = number of 7r 2 iterns correctly classified 
n 2M ~ number of 7r 2 items misclassified 
The apparent error rate is then 

APER = — - — (11-34) 

n x + n 2 v 

which is recognized as the proportion of items in the training set that are misclassified. 
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Example 11.6 (Calculating the apparent error rate) Consider the classification re¬ 
gions Ri and R 2 shown in Figure 11.1 for the riding-mower data. In this case, obser¬ 
vations northeast of the solid line are classified as 77 -;, mower owners; observations 
southwest of the solid line are classified as 7 t 2 , nonowners. Notice that some obser¬ 
vations are misclassified. The confusion matrix is 

Predicted membership 
7 ri: riding-mower owners 7r 2 : nonowners 
riding- ~ ~~ 

77 -]: mower n lC = 10 n iM = 2 n\ = 12 

owners _ 

tt 2 : nonowners n 2M ~ 2 n 2C = 10 n 2 = 12 

The apparent error rate, expressed as a percentage, is 

APER = Q * 100% = (^) 100% = 16.7% ■ 

The APER is intuitively appealing and easy to calculate. Unfortunately, it tends 
to underestimate the AER, and the problem does not disappear unless the sample 
sizes ni and n 2 are very large. Essentially, this optimistic estimate occurs because the 
data used to build the classification function are also used to evaluate it. 

Error-rate estimates can be constructed that are better than the apparent error 
rate, remain relatively easy to calculate, and do not require distributional assump¬ 
tions. One procedure is to split the total sample into a training sample and a valida¬ 
tion sample. The training sample is used to construct the classification function, and 
the validation sample is used to evaluate it.The error rate is determined by the pro¬ 
portion misclassified in the validation sample. Although this method overcomes the 
bias problem by not using the same data to both build and judge the classification 
function, it suffers from two main defects: 

(i) It requires large samples. 

(ii) The function evaluated is not the function of interest. Ultimately, almost all of 
the data must be used to construct the classification function. If not, valuable in¬ 
formation may be lost. 

A second approach that seems to work well is called Lachenbruch’s “holdout” 
procedure 7 (see also Lachenbruch and Mickey [24]): 

1. Start with the 77 ^ group of observations. Omit one observation from this 
group, and develop a classification function based on the remaining n\ — 1, n 2 
observations. 

2. Classify the “holdout” observation, using the function constructed in Step 1. 


Actual 

membership 


7 Lachenbruch’s holdout procedure is sometimes referred to as jackknifing or cross-validation. 
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3. Repeat Steps 1 and 2 until all of the 7 iq observations are classified. Let n[*$ he 
the number of holdout (H) observations misclassified in this group. 

(H) 

4. Repeat Steps 1 through 3 for the 7 t 2 observations. Let n 2 jJ be the number of 
holdout observations misclassified in this group. 

Estimates P(211) and P(112) of the conditional misclassification probabilities 
in ( 11 - 1 ) and ( 11 - 2 ) are then given by 


P(211) =■ 


P(112) 


(11*35) 


and the total proportion misclassified, (rzj^ + / 4 w)/( n i + n 2 )> is > f° r moderate 
samples, a nearly unbiased estimate of the expected actual error rate, P(AER). 


P(AER) = 


+ *(") 
n l M + n 2M 

n 1 + n 2 


(11-36) 


Lachenbruch’s holdout method is computationally feasible when used in con¬ 
junction with the linear classification statistics in (11-18) or (11-19). It is offered as 
an option in some readily available discriminant analysis computer programs. 

Example 11.7 Calculating an estimate of the error rate using the holdout procedure) 

We shall illustrate Lachenbruch’s holdout procedure and the calculation of error 
rate estimates for the equal costs and equal priors version of (11-18). Consider the 
following data matrices and descriptive statistics. (We shall assume that the 
rii - n 2 = 3 bivariate observations were selected randomly from two populations 
7Ti and 7 t 2 with a common covariance matrix.) 


Sl = 10 • 


2 -2 

-2 8 


The pooled covariance matrix is 


Spooled a ( 2 S, + 2 S 2 ) — 


1 -1 
-1 4 


Using Spiled, the rest of the data, and Rule (11-18) with equal costs and equal pri¬ 
ors, we may classify the sample observations. You may then verify (see Exercise 
11.19) that the confusion matrix is 
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Classify as: 


Thie population: 


Wl 

^2 

77-j 2 

1 

7 T 2 1 

2 


and consequently, 


APER(apparent error rate) = — = .33 

6 


Holding out the first observation x' H = [2,12] from Xi, we calculate 


Y _ 4 10 _ _ 3.5 

Ai *- [3 sj ; Xi//_ L 9 j 

The new pooled covariance matrix, S Hpooled , is 

l r -~ 


and lSi h — 


//.pooled ^ f^Sl// T 2S 2 ] ^ 


1 2.5 -1 
3 -1 10 


with inverse 8 


1 10 1 

8 1 2.5 


It is computationally quicker to classify the holdout observation x 1H on the basis 
of its squared distances from the group means x^ and x 2 . This procedure is equivalent 
to computing the value of the linear function y ~ a' H x H = (x 1H - x 2 )'S//,pooled x h 
and comparing it to the midpoint m H - \{x XH ~ * 2 ) S//,pooled (*1 H + * 2 )- [ See 
(11-19) and (11-20).] 

Thus with x’ H = [2,12] we have 

Squared distance from x 1H = (x H - x 1H )’Spooled (*// “ *i//) 


= [2 - 3.5 12 — 9] ■ 


1 10 1 


L " J 8 L 1 2.5 J L12 

Squared distance from x 2 = (x H - * 2 ) Spooled (*// “ * 2 ) 


2 —3.5 
2 -9 


[2-4 12-7]- 


i no i 

8 L 1 2.5 


2 -4 
12 -7 


= 10.3 


Since the distance from x H to x iH is smaller than the distance from x w to x 2 , we 
classify x H as a 77-1 observation. In this case, the classification is correct. 

If xh = [4,10] is withheld, x iH and S// )POO ]e d become 


2.51 

\H - and S //, pooled ~ 


1 16 4 

8 4 2.5 


8 A mairix identity due to Bartlett [3] allows for the quick calculation of pooled directly from Sp*^. 
Thus one does not have to recompute the inverse after withholding each observation. (See Exercise 11.20.) 
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We find that 


(x H - X iH ) S //.pooled ( x /f ~ X 1 h) - [4 -2.5 10 - 10] 


1 16 4 4 — 2.5 

8 4 2.5 J I 10 - 10 


(*// ~ x 2 )'S // 1 >pooled (x H - x 2 ) = [4 - 4 10 - 7] 


1 16 4 4 -4 

8 4 2.5 10 — 7 


and consequently, we would incorrectly assign x'y = [4, 10 ] to tt 2 . Holding out 
x^ = [3,8] leads to incorrectly assigning this observation to 7 r 2 as well. Thus, 

■IS - 2. 

Turning to the second group, suppose x' H = [ 5 , 7 ] is withheld.Then 

y -F 3 9 1 - -T 3 * 5 ! a -T* 5 ~ 2 ~ 

^2 h ~ 4 5 h x 2 H - J ; ancl 1S 2W - _2 g 


The new pooled covariance matrix is 


&H .pooled ~ o [2Si + lSfeff] 


1 2.5 -4 


3 -4 16 


with inverse 


We find that 


3 16 4 


24 4 2.5 


(Xtf x i) ^//.pooled( x // x i) [5 3 7 10] 24 4 25 


3 16 4 5 -3 


7-10 


(x H - *2 hYSh, pooled ( X H ~ x 2 h) ~ [5 “ 3.5 1 ~ 7]; 


3 16 4 5 - 3.5 


24 4 2.5 7 - 7 


and x^ = [5,7] is correctly assigned to 7 t 2 . 
When x' H = [3,9] is withheld, 


( x // x i) ^//.pooled ( x // x i) [3 3 9 10] 24 125 9 — 10 


3 f 10 13-3 


, _ 3 r io 111~ 3 - 4 - 5 i 

(*H - * 2 «) S w , pooled (*// - * 2 «) = [3 - 4i 9-6]^ J 25 9 _ 6 


and x// = [3,9] is incorrectly assigned to 77 ]. Finally, withholding x H — [4,5] leads 
to correctly classifying this observation as 7 t 2 . Thus, = 1- 
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An estimate of the expected actual error rate is provided by 


£(aer) = 


AH) (H) 
n lM + n 2 M 

n\ + n 2 


2 + 1 
3 + 3 


.5 


Hence, we see that the apparent error rate APER = .33 is an optimistic measure of 
performance. Of course, in practice, sample sizes are larger than those we have 
considered here, and the difference between APER and ^(AER) may not be as 
large. ■ 


If you are interested in pursuing the approaches to estimating classification 
error rates, see [23]. 

The next example illustrates a difficulty that can arise when the variance of the 
discriminant is not the same for both populations. 


Example 11.8 (Classifying Alaskan and Canadian salmon) The salmon fishery is a 
valuable resource for both the United States and Canada. Because it is a limited 
resource, it must be managed efficiently. Moreover, since more than one country is 
involved, problems must be solved equitably.That is,Alaskan commercial fishermen 
cannot catch too many Canadian salmon and vice versa. 

These fish have a remarkable life cycle. They are born in freshwater streams 
and after a year or two swim into the ocean. After a couple of years in salt water, 
they return to their place of birth to spawn and die. At the time they are about to 
return as mature fish, they are harvested while still in the ocean. To help regulate 
catches, samples of fish taken during the harvest must be identified as coming 
from Alaskan or Canadian waters. The fish carry some information about their 
birthplace in the growth rings on their scales. Typically, the rings associated with 
freshwater growth are smaller for the Alaskan-born than for the Canadian-born 
salmon. Table 11.2 gives the diameters of the growth ring regions, magnified 100 
times, where 

X\ — diameter of rings for the first-year freshwater growth 
(hundredths of an inch) 

X 2 = diameter of rings for the first-year marine growth 
(hundredths of an inch) 

In addition, females are coded as 1 and males are coded as 2. 

Training samples of sizes n\ = 50 Alaskan-bom and n 2 = 50 Canadian-born 
salmon yield the summary statistics 


429.660 V 


260.608 -188.093 
-188.093 1399.086 
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The data appear to satisfy the assumption of bivariate normal distributions (see 
Exercise 11.31), but the covariance matrices may differ. However, to illustrate a point 
concerning misclassification probabilities, we will use the linear classification procedure. 

The classification procedure, using equal costs and equal prior probabilities, 
yields the holdout estimated error rates 


Actual 7Ti: Alaskan 

membership Canadian 

based on the linear classification function [see (11-19) and (11-20)] 
w = y - tn = -5.54121 - .12839*! + .05194* 2 
There is some difference in the sample standard deviations of w for the two 


populations: 


Sample 

Sample 


n 

Mean 

Standard Deviation 

Alaskan 

50 

4.144 

3.253 

Canadian 

50 

-4.147 

2.450 


Although the overall error rate (7/100, or 7%) is quite low, there is an unfair¬ 
ness here. It is less likely that a Canadian-born salmon will be misclassified as 
Alaskan born, rather than vice versa. Figure 11.8, which shows the two normal 
densities for the linear discriminant y, explains this phenomenon. Use of the 



?2 m y ] 


Predicted membership 
TTj: Alaskan 7? 2 : Canadian 


44 

6 

1 

49 


Figure I 1.8 Schematic of normal densities for linear discriminant—salmon data. 
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midpoint between the two sample means does not make the two m^classification 
probabilities equal. It dearly penalizes the population with the largest variance. 
Thus, blind adherence to the linear classification procedure can be unwise. m 

It should be intuitively clear that good classification (low error rates) will de¬ 
pend upon the separation of the populations. The farther apart the groups, the more 
likely it is that a useful classification rule can be developed. This separative goal, al¬ 
luded to in Section 11.1, is explored further in Section 11.6. 

As we shall see, allocation rules appropriate for the case involving equal prior 
probabilities and equal misclassification costs correspond to functions designed to 
maximally separate populations. It is in this situation that we begin to lose the dis¬ 
tinction between classification and separation. 

11.5 Classification with Several Populations 

In theory, the generalization of classification procedures from 2 to g a 2 groups is 
straightforward. However, not much is known about the properties of the corre¬ 
sponding sample classification functions, and in particular, their error rates have not 
been fully investigated. 

The “robustness” of the two group linear classification statistics to, for instance, 
unequal covariances or nonnoiinal distributions can be studied with computer gen¬ 
erated sampling experiments. 9 For more than two populations, this approach does 
not lead to general conclusions, because the properties depend on where the popu¬ 
lations are located, and there are far too many configurations to study conveniently. 

As before, our approach in this section will be to develop the theoretically opti¬ 
mal rules and then indicate the modifications required for real-world applications. 

The Minimum Expected Cost of Misclassification Method 

Let /,(x) be the density associated with population 777 , i = 1,2,... , g. [For the most 
part/we shall take f(x) to be a multivariate normal density, but this is unnecessary 
for the development of the general theory.] Let 

p. = the prior probability of population TT n i = 1,2,..., g 
c( k\i) = the cost of allocating an item to tt* when, in fact, it belongs 
to 7 Tj, fork, / - 1,2,... t g 

For k = /, c(i 1i) = 0. Finally, let R k be the set of x’s classified as ir k and 

P(k\i) = ^(classifyingitem as ir k I w,-) = J^fi(x)dx 
8 

for k,i = 1,2,..., g with P(i I/) = I ~ 

k~ i 
k*i 

9 Here robustness refen to the deterioration in error rates caused by using a classification procedure 
with data that do not conform to the assumptions on which the procedure was based. 

It is very difficult to study the robustness of classification procedures analytically. However, data 
from a wide variety of distributions with different covariance structures can be easily generated 
on a computer. The performance of various classification rules can then be evaluated using computer- 
generated “samples” from these distributions. 
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The conditional expected cost of misclassifying an x from tti into 7r 2 


ECM(l) = P(2I l)c(2l 1) + P(31 l)c(311) +•■• + P(g\l)c(gl - 
= jz P(k\l)c(k\l) 

k=2 

This conditional expected cost occurs with prior probability p x , the probab*** i- 
In a similar manner, we can obtain the conditional expected costs of 
cation ECM(2),..., ECM(g). Multiplying each conditional ECM by its 
ability and summing gives the overall ECM: 

ECM = p!ECM(l) + p2ECM(2) + ■■■ + p g ECM(g) 


= Pi['kP(k\mk\l)) +/J j? P(k\2)c(k\2) 

\k=2 / \ k =1 


+ *'* + Pg\ z P(k\g)c(k\g) 


= 2 Pi ^ p {k\i)c(k\i) 


,=1 / 

Determining an optimal classification procedure amounts to choosing ~£- 
tually exclusive and exhaustive classification regions R x , ..., R g su 

(11-37) is a minimum. 

Result 11.5. The classification regions that minimize the ECM (11-37) are ^ ^ 
by allocating x to that population 7 ?>, k = 1,2, . . ., g, for which 


^ Pifi(*)c(k\i) 


is smallest. If a tie occurs, x can be assigned to any of the tied populations. 
Proof. See Anderson [2]. 

Suppose all the misclassification costs are equal, in which case the minimum exp: 
cost of misclassification rule is the minimum total probability of misclassific 
rule. (Without loss of generality, we can set all the misclassification costs equal 
Using the argument leading to (11-38), we would allocate x to that popuJ 
7 T k ,k — 1,2,..., g y for which 


2 Pifii* 


a 
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is smallest. Now, (11-39) will be smallest when the omitted term, p k fk( x ), is largest. 
Consequently, when the misclassification costs are the same, the minimum expected 
cost of misclassification rule has the following rather simple form. 


Minimum ECM Classification Rule 
with Equal Misclassification Costs 

Allocate x 0 to ir k if 

Pkfk(x) > P,7i(*) for 311/ * k 


or, equivalently. 
Allocate x 0 to ir k if 


In Pkfki*) > In Pifi(x) for all i * k 


(11-40) 


(11-41) 


It is interesting to note that the classification rule in (11-40) is identical to the 
one that maximizes the “posterior” probability P(t 7 *!x) = P (x comes from ir k 
given that x was observed), where 


, „ P*/*(*) (prior) x (likelihood) 

P(TT k \x) =—- = --- 

2 PtM x ) 21 (prior) X (likelihood)] 


for A: = 


(11-42) 

Equation (11-42) is the generalization of Equation (11-9) to g ^ 2 groups. 

You should keep in mind that, in general, the minimum ECM rules have three 
components: prior probabilities, misclassification costs, and density functions. These 
components must be specified (or estimated) before the rules can be implemented. 


Example 11.9 (Classifying a new observation into one of three known populations) 

Let us assign an observation x 0 to one of the g - 3 populations i r*, 7 r 2 , or 7 t 3 , given 
the following hypothetical prior probabilities, misclassification costs, and density 
values: 

True population 



n 

n 

^3 


^1 

c(l|l) =0 

c( 1|2) = 500 

c(l 1 3) = 100 . 

Classify as: 

7T 2 

c( 211 ) = 10 

c( 212 ) = 0 

c(2|3) = 50 


”3 

c(311) = 50 

c(3 12 ) = 200 

c (3 1 3) = 0 

Prior probabilities: 


ii 

S 

Pi = .60 

P3 = -35 

Densities at x 0 : 


/i(x 0 ) = .01 

/ 2 (x„) = .85 

/3(*o) = 2 


We shall use the minimum ECM procedures. 
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The values of ]£ Pif,{x 0 )c(k I/) [see (11-38)] are 
1 = 1 
i^k 

k = 1: 112) + p 3 / 3 (x 0 )c(l 13) 

= (.60) (.85) (500) + (.35) (2) ( 100 ) = 325 

k = 2: Pi/i(*o)c(2H) + p 3 f 3 (x 0 )c(2\3) 

= (.05) (.01) ( 10 ) + (.35) (2) (50) = 35.055 
k = 3: Pi/i(*o)c(3l 1) + P 2 / 2 (x 0 )c( 3 l 2 ) 

= (.05) (.01) (50) + (.60) (.85) ( 200 ) = 102.025 

3 

Since 2 Pifi( x o) c ( k 10 is smallest for k = 2, we would allocate Xq to 7 r 2 . 


If all costs of misclassification were equal, we would assign Xq according to 
(11-40), which requires only the products 

Pi/i(*o) = (.05) (.01) = .0005 

PtS\ r z(*o) = (-60) (.85) = .510 

P3/3(x 0 ) = (.35) (2) = .700 

Since 

P3f 3 (x 0 ) = .700 a Pifi{x 0 ),i = 1,2 

we should allocate x 0 to 7 t 3 . Equivalently, calculating the posterior probabilities [see 
(11-42)], we obtain 


p (^i l*o) 


2 PiM* 0 ) 


(.05) (.01) 


= .0004 


(.05) (. 01 ) + (.60) (.85) + (.35) (2) " 1.2105 “ 

p (« 2 i*o) = = imm , ^io_ = 

v t, s 12105 1.2105 1 

2 p-M* 0 ) 

i=l 

P(n 3 l*o) = - 3 P3/3(Xo) = 1 3 ^l = J*L = 578 

2 , 1.2105 1.2105 

2 Pi/i(*o) 

1 = 1 

We see that x 0 is allocated to v 3 , the population with the largest posterior probability. 

Classification with Normal Populations 

An important special case occurs when the 


m = 


(27r)P/ 2 |S,r /I ' 


exp -~(x - (id% l (x - fti) , 


/=l,2,...,g (11-43) 
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are multivariate normal densities with mean vectors /i, and covariance matrices X,-. 

If, further, c(i I i) = 0,c(fcl/) = 1, k * i (or, equivalently, the misclgssification costs 
are all equal), then (11-41) becomes 

Allocate x to n k if 

In Pkfk( x ) ~ ~~ 2 2^ X ~ ^ ~ ^ 

= max In /?//)(x) (11-44) 

i 

The constant (p/2) In (2n) can be ignored in (11-44), since it is the same for all 
populations. We therefore define the quadratic discrimination score for the ith 
population to be 

d?(x) = — § In | X, | - f(x - M.O'X'H* “ Mi) + In# 

i = 1,2,..., g (11-45) 

The quadratic score d?(x) is composed of contributions from the generalized 
variance | X-, |, the prior probability p, , and the square of the distance from x to the 
population mean p,. Note, however, that a different distance function, with a 
different orientation and size of the constant-distance ellipsoid, must be used for 
each population. 

Using discriminant scores, we find that the classification rule (11-44) becomes 
the following: 

Minimum Total Probability of Misclassification (TPM) Rule 
for Normal Populations—Unequal 2,- 

Allocate x to 7 r k if 

the quadratic score d® (x) = largest of df(x) t d^(x),... ,d^(x) (11-46) 

where df(x) is given by (11-45). 

In practice, the /t, and Xi are unknown, but a training set of correctly classified 
observations is often available for the construction of estimates. The relevant sam¬ 
ple quantities for population Vi are 

x ( . = sample mean vector 
S* = sample covariance matrix 

and 

n t - sample size 

The estimate of the quadratic discrimination score d, (x) is then 

d?(x) = -|ln|S,| - j(x - SjjVt* - x,) + lnp„ i= 1,2,...,g (H- 47 ) ; 
and the classification rule based on the sample is as follows: 
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Estimated Minimum (TPM) Rule 
for Several Normal Populations—Unequal 2, 

Allocate x to tt* if 

the quadratic score d k (x) = largest of df(x) y £2 (x),... 4®(x) (11-48) 

where ^p(x) is given by (11-47). 


A simplification is possible if the population covariance matrices, X h are equal. 
When Xi = S, for / = 1,2,..., g, the discriminant score in (11-45) becomes 

<*?(.*) = —|ln|X| - jx'2 _1 x + n'iX^x - + In# 

The first two terms are the same for d¥(x),d$(x), .... df(x), and, consequently, 
they can be ignored for allocative purposes. The remaining terms consist of a con¬ 
stant a = In pi - and a linear combination of the components of x. 

Next, define the linear discriminant score 

4(x) = fi-X-'x - + In Pi (11-49) 

fort = l,2,...,g 

An estimate d,(x) of the linear discriminant score d,(x) is based on the pooled 
estimate of X. 

Spooled = . 1 , --((«1 - 1)S, + (« 2 “ 1)®2 + ■■• + («,- l)S f ) 

n \ + n 2 + * ■ * + n g - g 

(11-50) 

and is given by 

dj(x) = xJSpooiedx - jx'S^cedx,- + In p, (11-51) 

fori = 1,2,... ,g 


Consequently, we have the following: 


Estimated Minimum TPM Rule 
for Equal-Covariance Normal Populations 

Allocate x to v k if 

the linear discriminant score d k (x) = the largest of di(x), d 2 (x),..., d g (x) 

(11-52) 

with di{x) given by (11-51). 


Comment. Expression (11A9) is a convenient linear function of x. An equivalent 
classifier for the equal-covariance case can be obtained from (11-45) by ignoring the 
constant term, — § In | S |- The result, with sample estimates inserted for unknown 
population quantities, can then be interpreted in terms of the squared distances 

D 2 M = (x - x,/'Spooled (x - X,) (11-53) 
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from x to the sample mean vector x,. The allocatory rule is then 

Assign x to the population 7r t for which - |l>?(x) + In is largest (11-54) 

We see that this rule—or, equivalently, (11-52)—assigns x to the “closest” popula¬ 
tion. (The distance measure is penalized by In p x .) 

If the prior probabilities are unknown, the usual procedure is to set p l - p 2 = ... ^ 
p g = 1/g. An observation is then assigned to the closest population. 




52S^o,edS 2 = ^[48 39] ^ =~ 

and 

«*>-■"(■*> + (§)<■■ + (I)*»-Kf) 

Finally, 

« S ^-[0 - 2 ] i [“ |] - i [-6 - 18 ] 

•SSpi^aS. - 5J[ -6 

and 

- In(.50) + (g),„ + (^)x 0! - i(ff) 
Substituting the numerical values jc 01 = -2 and x 02 ~ -1 gives 

- -1.3S6 + (^)( -2 ) + (M)( _1) ~ft> 

iw--1.386.(1)(-2) .(§)<-!> -f- -8.158 

*<*»>-- J6,3 + ( ir )<- 2 > + ( ir )<-‘»-1 *- 35 » 


Since d 3 (x 0 ) ~ -.350 is the largest discriminant score, we allocate x 0 to tt 3 . m 
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Example 11.11 (Classifying a potential business-school graduate student) The ad¬ 
mission officer of a business school has used an “index” of undergraduate 
grade point average (GPA) and graduate management aptitude test (GMAT) 
scores to help decide which applicants should be admitted to the school’s gradu¬ 
ate programs. Figure 11.9 shows pairs of x 2 = GPA, x 2 = GMAT values for 
groups of recent applicants who have been categorized as ir : : admit; ir 2 : do not 
admit- and 7 r 3 : borderline. 10 The data pictured are listed in Table 11.6. (See 
Exercise 11.29.) These data yield (see the SAS statistical software output in 
Panel 11.1) 

m = 31 n 2 = 28 "3 = 26 


_ r 3.40 - 

Xl " |_561.23_ 



f 2.99 _ 
“ |_446.23_ 


2.97 
488.45 _ 


.0361 -2.0188 

-2.0188 3655.9011 


GMAT 



2.10 2.40 2.70 3.00 3.30 3.60 



Admit (Tij) 

Do not admit (te 2 ) 
Borderline (rc 3 ) 



Figure 11.9 Scatter plot of (x 2 = GPA,* 2 = GMAT) for app'^^^^ft 6 
school of business who have been classified as admit, do not admit, or bor 


lc Tn this case, the populations are artificial in the sense that they haveWo^GPA an^high 
admissions officer. On the other hand, experience has shown that applicants with 8 h variables 
GMAT scores generally do well in a graduate program; those with low readings 0 
generally experience difficulty. 
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Suppose a new applicant has an undergraduate GPA of Jtj = 3.21 and a GMAT 
score of x 2 = 497. Let us classify this applicant using the rule in (11-54) with equal 
prior probabilities. 

With Xq - [3.21,497], the sample squared distances are 


r>l(*o) = (x 0 - *l)'Spioled(*0 - *l) 

= [3.21 - 3.40, 497 - 561.23] 


28.6096 .0158 


3.21 - 3.40 


0158 .0003 497 - 561.23 


■Di(*o) = (*o “ x 2 )'Spi oled (x 0 - x 2 ) = 17.10 

D|(x 0 ) = (x 0 - x 3 )'S;i,ied(*o - * 3 ) = 2.47 

Since the distance from Xq = [3.21,497] to the group mean x 3 is smallest, we assign 
this applicant to 7 t 3 , borderline. ■ 

The linear discriminant scores (11-49) can be compared, two at a time. Using 
these quantities, we see that the condition that d k (x) is the largest linear discrimi¬ 
nant score among d x (x), d 2 (x ),..., d g (x) is equivalent to 

0 ^ d k (x) - d t (x) 


= (Hk - M;)'£ _i x - -(m* - /»/)'£ V* + Mi) + to 
for all i ~ 1, 2,..., g. 


PANEL I U SAS ANALYSIS FOR ADMISSION DATA USING PROC DISCRIM. 

title 'Discriminant Analysis'; 

data gpa; 

infile 'T11-6.dat'; 

input gpa gmat admit $; 

proc discrim data = gpa 

method = normal pool = yes manova wcov pcov listerr crosslisterr; 
priors 'admit' = .3333 'notadmit' = .3333 'border' ^ .3333; 
class admit; var gpa gmat; 


PROGRAM COMMANDS 


DISCRIMINANT ANALYSIS 
85 Observations 84 DF Total 

2 Variables 82 DF Within Classes 

3 Classes 2 DF Between Classes 

Class Level Information 


OUTPUT 


ADMIT 

adfpit 

border 

notadmit 


Frequency 

31 

26 

28 


Weight 

31.0000 

26.0000 

28.0000 


Proportion 

0.364706 

0.305882 

0.329412 


■ - .P 

Probability 

0.333333 

0.333333 

0.333333 


(continues on next page) 
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PANEL! M (continued) 


DISCRIMINANT ANALYSIS WITHIN-CLASS COVARIANCE MATRICES 
ADMIT = admit DF = 30 

Variable GPA GMAT 

GPA 0.043558 0.058097 

GMAT 0.05BQ97 4618.247312 


ADMIT = border 
Variable GPA 

GPA 0.029692 

GMAT -5.403846 


DF = 25 

GMAT 

-5.403846 

2246.904615 


ADMIT = notadmit 
Variable GPA 

GPA 0.033649 

GMAT -1.192037 


DF = 27 

GMAT 

-1.192037 

3891.253968 


Pooled Within-Class Covariance Matrix DF. = 82 


0.036068 

-2.018759 


-2,018759 

3655.901121 


Statistic 
Wilks' Lambda 
pillai's Trace 
Hotelling-Lawley Trace 
Roy's Greatest Root 


Multivariate Statistics and F Approximations 
S = 2 M = -0.5 N = 39.5 
Value F Num DF Den 

0.12637661 73.4257 4 1 

1.00963002 41.7973 4 1 

5.83665601 116.7331 4 

5.64604452 231.4878 2 

NOTE: F Statistic for Roy's Greatest Root is an upper bound. 
NOTE: F Statistic for Wilks' Lambda is exact. 


DISCRIMINANT ANALYSIS LINEAR DISCRIMINANT FUNCTION ^ 
Constant = - .5X- COXT 1 Xj + In PRIORj Coefficient Vector = COV' 1 X, 


CONSTANT 

GPA 

GMAT 


ADMIT 

admit 

-241.47030 

106.24991 

0.21218 


border 

-178.41437 

92.66953 

0.17323 


notadmit 

-134.99753 

78.08637 

0.16541 


Classification Results for Calibration ■ 

Resubstitptiph Results u sing Linear ^iscrintindnt Function. 

Generalized Squared Distance Function: 

D?(X) = (X - Xj)'cov" 1 (X - Xj) 

Posterior Probability of Membership in each ADMIT: 
Pr(jlX) = exp(-.5Df(X))/5UMexp(-.5D 2 k (X)) 
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PANEL 1 l.l (continued) 


Posterior Probability of Membership in ADMIT: 


Obs 

From 

Classified 






ADMIT 

into ADMIT 


admit 

border 

notadmit 

2 

admit 

border 

* 

0.1202 

0.8778 

0.0020 

3 

admit 

border 

* 

0.3654 

0.6342 

0.0004 

24 

admit 

border 

* 

0.4766 

0.5234 

0.0000 

31 

admit 

border 

* 

0.2964 

0.7032 

0.0004 

58 

notadmit 

border 

* 

0.0001 

0.7550 

0.2450 

59 

notadmit 

border 

* 

0.0001 

0.8673 

0.1326 

66 

border 

admit 

* 

0.5336 

0.4664 

0.0000 


*Misdassified observation 

Classification Summary for Calibration Data: WORK.GPA 
Cross validation Summary using Linear Discriminant Function 

Generalized Squared Distance Function: 

D?(X) = (X - X, x)i )'COV^(X - X, x)l ) 

Posterior Probability of Membership in each ADMIT: 
Pr(jlX) - exp(-.5Df(X))/SUMexp(-.5Di(X)) 

Number of Observations and Percent Classified into ADMIT: 


Total 

Percent 

Priors 


Error Count Estimates for ADMIT: 
admit border notadmit 

0.1613 0.0769 0.0714 

0.3333 0.3333 0.3333 


Adding -ln(pjt/pj) = ln(p,/p*) to both sides of the preceding inequality gives 
the alternative form of the classification rule that minimizes the total probability of 
misclassification. Thus, we 
Allocate x to ir k if 


Mi)'S x Mi)'S (M* + Mi) £= In ~ 


for all/ = 1,2,... ,g. 


(11-55) 
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Now denote the left-hand side of (11-55) by d ki (x). Then the conditions in 
(11-55) define classification regions R„R 2 ,...,R g , which are separated by (hyper) 
planes. This follows because d ki (x) is a linear combination of the components of x 
For example, when g = 3, the classification region Ri consists of all x satisfying 


R\:d u (x) ^ ln^— 


fori = 2,3 


That is, Ri consists of those x for which 

d 12 (x) = (Ml - M2)'2 _1 x - ^(Mi - M2)'2 _1 (Mi + M 2 ) 2= 


and, simultaneously , 


d 13 (x) = (Mi " M3)'2 T’x 


j(Mi - M3)'2 Vi + Ms) s In ^ 


Assuming that and M 3 do not lie along a straight line the equations d, 2 (x) = 

in (ft/p,) and d 13 (x) = ln(p 3 /Pi) define two intersecting hypeiplanes that delin¬ 
eate R in the p-dimensional variable space. The term ln(ft/pi) places the plane 
closer to Mi than p. 2 if ft is greater than ft . The regions R U R 2 , and R 3 are shown m 
Figure 11.10 for the case of two variables. The picture is the same for more variables 
if we graph the plane that contains the three mean vectors. 

The sample version of the alternative form in (11-55) is obtained by substituting 
Xj for m, and inserting the pooled sample covariance matrix S p00 | ed for S. When 

2) («, - 1) a P, so that S^ 0 ,ed exists, this sample analog becomes 



2 4 6 8 


Figure 11.10 The classification 
regions Ri , R 2 , and R$ for the 
linear minimum TPM rule 
(Pl = J-P2 = J.ft = i)’ 
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Allocate x to ir k if 

d*i(x) = (x* - x,)'S^o, ed x - ^(x* - X,)'S,^ 0 |e d (X* + X,) 

a ln^-^-J for all i * k (11-56) 

Given the fixed training set values xj and S^jed, d ki (x) is a linear function of 
the components of x. Therefore, the classification regions defined by (11-56)—or, 
equivalently, by (11-52)—are also bounded by hyperplanes, as in Figure 11.10. 

As with the sample linear discriminant rule of (11-52), if the prior probabilities 
are difficult to assess, they are frequently all taken to be equal. In this case, 
In (Pi/Pk) = 0 for all pairs. 

Because they employ estimates of population parameters, the sample classifi¬ 
cation rules (11-48) and (11-52) may no longer be optimal. Their performance, 
however, can be evaluated using Lachenbruch’s holdout procedure. If »iM is the 
number of misclassified holdout observations in the ith group, i = 1,2,..., g, then 
an estimate of the expected actual error rate, £(AER), is provided by 

& n {H) 
n iM 

£(AER) = - (11-57) 

2* 

i- 1 


Example II. 12 (Effective classification with fewer variables) In his pioneering work 
on discriminant functions, Fisher [9] presented an analysis of data collected by 
Anderson [1] on three species of iris flowers. (See Table 11.5, Exercise 11.27.) 

Let the classes be defined as 

7 t i: Iris setosa; 7r 2 : Iris versicolor ; 7r 3 : Iris virginica 

The following four variables were measured from 50 plants of each species. 

X\ = sepal length, X 2 = sepal width 
X$ = petal length, X 4 = petal width 

Using all the data in Table 11.5, a linear discriminant analysis produced the confusion 
matrix 

Predicted membership 


7r*: Setosa 

Actual 

membership 71 ^ Versicolor 

7t 3 : Virginica 
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The elements in this matrix were generated using the holdout procedure, so . 
(see 11-57) j 

e(aer) = Tio = - 02 ■ 


The error rate,2%, is low. , ... , . 

Often it is possible to achieve effective classification with fewer variables* It is- 
oood practice to try all the variables one at a time, two at a time, three at a time and; 
so forth, t<? see how well they classify compared to the discriminant function, which; 

i.cac all the variables. . 4 „ . . ~ 

If we adopt the holdout estimate of the expected AER as our cntenon, we find 

for the data on irises: 


Single variable 

Misclassification rate 

~ Xi 

.253 

x 2 

.480 

*3 

.053 

*4 

.040 

Pairs of variables 

Misclassification rate 


X, X 2 

y\ v, .040 

x xl .040 4-if 

xfx, Ml 

tx\ I S3 £ 

* r to to«Ts£ieS'to 

gu w e 8 s Box plots of Z 4 = petal width are shown in Figure 11.11 for the three 
Pet3 DaiToch^and Mosimann [6] have suggested that these species °^ ris ^ y 

dure classifies its target objects. ^ 

O- dto-to to total to .rnptote to 

or (11-56), and many commercial computer programs remember that' id S ? 

the linear discriminant rule has a simple s ““^ e ’ y ltivaliate normality and||J 
was derived under the rather strong assumptions of mu ti tentative^ “ 

Mual covariances. Before implementing a linear classification rule, these s 



Fisher’s Method for Discriminating among Several Populations 621 



Figure 11.11 Box plots of petal width for the three species of iris. 


assumptions should be checked in the order multivariate normality and then equal¬ 
ity of covariances. If one or both of these assumptions is violated, improved classifi¬ 
cation may be possible if the data are first suitably transformed. 

The quadratic rules are an alternative to classification with linear discriminant 
functions. They are appropriate if normality appears to hold, but the assumption of 
equal covariance matrices is seriously violated. However, the assumption of normal¬ 
ity seems to be more critical for quadratic rules than linear rule§. If doubt exists as to 
the appropriateness of a linear or quadratic rule, both rules can be constructed and 
their error rates examined using Lachenbruch’s holdout procedure. 


11.6 Fisher’s Method for Discriminating 
among Several Populations 

Fisher also proposed an extension of his discriminant method, discussed in 
Section 11.3, to several populations. The motivation behind the Fisher discriminant 
analysis is the need to obtain a reasonable representation of the populations that in¬ 
volves only a few linear combinations of the observations, such as aix, a^x, and a^x. 
His approach has several advantages when one is interested in separating several 
populations for (1) visual inspection or (2) graphical descriptive purposes. It allows 
for the following: 

1. Convenient representations of the g populations that reduce the dimension 
from a very large number of characteristics to a relatively few linear combina¬ 
tions. Of course, some information—needed for optimal classification—may be 
lost, unless the population means lie completely in the lower dimensional space 
selected. 
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2. Plotting of the means of the first two or three linear combinations (discriminants^ 
This helps display the relationships and possible groupings of the populations. %| 

3. Scatter plots of the sample values of the first two discriminants, which can indi- * 
cate outliers or other abnormalities in the data. 

The primary purpose of Fisher’s discriminant analysis is to separate populations. It 
can, however, also be used to classify, and we shall indicate this use. It is not neces4^ 
sary to assume that the g populations are multivariate normal. However, we do 
assume that the pXp population covariance matrices are equal and of full rank. 1 - 
That is, Xi = X 2 “ • • • “ Xg ~ X. 

Let ji denote the mean vector of the combined populations and the between 
groups sums of cross products, so that • ^ 


- 2 (Mi - m)(M/ - M)' 

r=1 

We consider the linear combination 


which has expected value 

E(Y) = a'£(Xf TTi) = a>, 


where /jl = — ^ /*/ 

8 i=i 


for population 7ri 


and variance 


Var(F) = a' Cov(X)a = a'Xa for all populations 


Consequently, the expected value p iY = a' /a, changes as the population from which 
X is selected changes. We first define the overall mean 

1 * 1 * ,(\ * \ 

My = T 2 MiV = ~ 2 a M, = a ( ~ 2 Mi J 

6 i=l 6 i=l Vo i=l / 


and form the ratio 




(variance of F) 


\i=i_ 


£)(Mi - m)' j a 


2 (M*y - Mr) 2 aB 


(11-59) 


11 If not, we let P = [ej,..., e fl ] be the eigenvectors of X corresponding to nonzero eigenvalue* 
[A 1( ..., aJ. Then we replace X by P'X which has a full rank covariance matrix P'XP. 
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The ratio in (11-59) measures the variability between the groups of F-values relative 
to the common variability within groups. We can then select a to maximize this ratio. 

Ordinarily, X and the /a, are unavailable, but we have a training set consisting of 
correctly classified observations. Suppose the training set consists of a random sam¬ 
ple of size n t from population ir h i = 1,2,..., g. Denote the n { X p data set, from 
population , by X* and its jth row by xj y . After first constructing the sample mean 
vectors 

1 Hl 

x, = — 2 x iy 

n i y = l 

and the covariance matrices S/, i = l,2,...,g, we define the “overall average” 
vector 

1 8 

5 4 2*, 

6 1 = 1 

which is the p X 1 vector average of the individual sample averages. 

Next, analagous to we define the sample between groups matrix B. Let 

B = 2 (*< "*) (*i ~ *)' (11-60) 

1=1 

Also, an estimate of X is based on the sample within groups matrix 

w = 2 K - 1) S - = 2 2 (*iy - */)(*./ - *i)' (11-61) 

i=l i=l /=l 

Consequently, W/(«! + n 2 + ■ •• + n g - g) = S pooled is the estimate of X. Be¬ 
fore presenting the sample discriminants, we note that W is the constant 
( n } + n 2 + •• * + n g - g) times so the same a that maximizes 

a'Ba/a'Spooieja also maximizes a'Ba/a'Wa. Moreover, we can present the optimiz¬ 
ing a in the more customary form as eigenvectors e, of W -1 B, because if 
W^Be = Ae then S^^Be = \(n ^ + n 2 + • • • + n g - g)e. 


Fisher’s Sample Linear Discriminants 

Let A 1? A 2 ,...,A 5 > 0 denote the s ^ min(g - l,p) nonzero eigenvalues of 
W and be the corresponding eigenvectors (scaled so that 

£'Spooi ed e = l).Then the vector of coefficients a that maximizes the ratio 

i'Bi •'(iw-*)®-*)')* 

-S- <■■«> 

a ' 2 2 (*// “ */) (*// - *,)' a 

L#«i J 

is given by a! = The linear combination is, called the sample first dis¬ 
criminant. The choice a 2 = e 2 produces the sample second discriminant , a 2 x, and 
continuing, we obtain a*x = the sample kth discriminant , k < s. 
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Exercise 11.21 outlines the derivation of the Fisher discriminants. The discriminants 
will not have zero covariance for each random sample X,. Rather, the condition 


„ fl if i = k^s 

a, Spooled a* j 0 otherw j se 


(11-63) 


will be satisfied. The use of is appropriate because we tentatively assumed 

that the g population covariance matrices were equal. 

Example 11.13 (Calculating Fisher's sample discriminants for three populations) 

Consider the observations on p = 2 variables from g = 3 populations given in 
Example 11.10. Assuming that the populations have a common covariance matrix 
2, let us obtain the Fisher discriminants.The data are 


tti (ni = 3) 

"-2 5~ 
Xi = 0 3; 

-1 1 


^2 («2 = 3 ) 


^3 («3 = 3 ) 

1 -2 
X 3 = 0 0 


In Example 11.10, we found that 

= _r-ii 


B = 2(*. - *)(*■' “ *) 

i=l 


1 62/3 


(*<7 “ x <)(*<7 ~ *<)' = ( n i +n 2 + n 3 3 ) s p 


6 -2 
-2 24 _ 

1 |~24 2 


140 2 6 


.3571 .4667 
.0714 .9000 


To solve for the s £ min(g - 1,|>) = min(2,2) = 2 nonzero eigenvalues of W 'B, 
we must solve 


IW -1 B - All 


.3571 - A 
.0714 


.4667 
.9000 - A 


(.3571 - A) (.9000 - A) - (.4667) (.0714) = A 2 - 12571A + .2881 - 0 

Using the quadratic formula, we find that Aj = .9556 and A 2 = .3015. The nor 
malized eigenvectors a, and Sj are obtained by solving 

• (W -, B - Ail)aj = 0 i = 1.2 
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and scaling the results such that a;S poo j e d£/ = 1. For example, the solution of 


(W -1 B - AjI) aj 


.3571 - .9556 .4667 


.0714 .9000 - .9556 I I a. 


ail I 0 


is,after the normalization aJSpooiedai = 1, 

ai = [386 .495] 


Similarly, 


The two discriminants are 


ai = [.938 -.112] 


>! = aix = [386 .495] | 1 I = 386jc! + .495jc 2 


>*2 = a 2 x = [-938 -.112] 1 = .938^ - .112x 2 

X2_\ 


Example 11.14 (Fisher's discriminants for the crude-oil data) Gerrild and Lantz [13] 
collected crude-oil samples from sandstone in the Elk Hills, California, petroleum 
reserve. These crude oils can be assigned to one of the three stratigraphic units 
(populations) 

7Ti'. Wilhelm sandstone 
tt 2 : Sub-Mulinia sandstone 
7 r 3 : Upper sandstone 

on the basis of their chemistry. For illustrative purposes, we consider only the five 
variables: 

X\ = vanadium (in percent ash) 


X 2 = Viron (in percent ash) 


X 3 = Vberyllium (in percent ash) 

X 4 ~ l/[saturated hydrocarbons (in percent area)] 

X 5 = aromatic hydrocarbons (in percent area) 

The first three variables are trace elements, and the last two are determined from 
a segment of the curve produced by a gas chromatograph chemical analysis. Table 
1L7 (see Exercise 11.30) gives the values of the five original variables (vanadium, 
iron, beryllium, saturated hydrocarbons, and aromatic hydrocarbons) for 56 cases 
whose population assignment was certain. 

A computer calculation yields the summary statistics 
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(38 + 11 + 7 — 3) Spooled 

'187.575 

1.957 41.789 

W = -4.031 2.128 3.580 

1.092 -.143 -.284 .077 

79.672 -28.243 2.559 -.9% 338.023 

There are at most 5 = min(g - l,p) = min (2,5) = 2 positive eigenvalues of 
W ‘b, and they are 4.354 and .559.The centered Fisher linear discriminants are 

ft = .312 (jc, - 6.180) - .710(^2 - 5.081) + 2.764(x 3 - .511) 

+ 11.809(x 4 - .201) - -235(x 5 - 6.434) 
ft = .169 (jc, - 6.180) - .245(x z - 5.081) - 2.046(x 3 - .511) 

— 24.453(x 4 — .201) - .378(x 5 — 6.434) 

The separation of the three group means is fully explained in the two- 
dimensional “discriminant space.”The group means and the scatter of the mdividhaal 
observations in the discriminant coordinate system are shown in Figure 11.12. llie 
separation is quite good. * 


and 

(rij + n 2 + n 3 - 3)8^0^ = 



Figure 11.12 Crude-oil samples in discriminant space. 
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Example 11.15 (Plotting sports data in two-dimensional discriminant space) Investi¬ 
gators interested in sports psychology administered the Minnesota Multiphasic 
Personality Inventory (MMPI) to 670 letter winners at the University of Wisconsin 
in Madison. The sports involved and the coefficients in the two discriminant 
functions are given in Table 11.3. 

A plot of the group means using the first two discriminant scores is shown in 
Figure 11.13. Here the separation on the basis of the MMPI scores is not good, 
although a test for the equality of means is significant at the 5% level. (This is due to 
the large sample sizes.) 

While the discriminant coefficients suggest that the first discriminant is most 
closely related to the L and Pa scales, and the second discriminant is most closely 
associated with the D and Pt scales, we will give the interpretation provided by the 
investigators. 

The first discriminant, which accounted for 34.4% of the common variance, was 
highly correlated with the Mf scale (r = -.78). The second discriminant, which 
accounted for an additional 18.3% of the variance, was most highly related to scores 
on the Sc, F, and D scales (r's = .66, .54, and .50, respectively). The investigators 
suggest that the first discriminant best represents an interest dimension; the second 
discriminant reflects psychological adjustment. 

Ideally, the standardized discriminant function coefficients should be examined 
to assess the importance of a variable in the presence of other variables. (See [29].) 
Correlation coefficients indicate only how each variable by itself distinguishes the 
groups, ignoring the contributions of the other variables. Unfortunately, in this case, 
the standardized discriminant coefficients were unavailable. 

In general, plots should also be made of other pairs of the first few discrimi¬ 
nants. In addition, scatter plots of the discriminant scores for pairs of discriminants 
can be made for each sport. Under the assumption of multivariate normality, the 


Table 11.3 

Sport 

Sample size 

MMPI 

Scale 

First 

discriminant 

Second 

discriminant 



QE 

.055 

-.098 

Football 

158 

L 

—.194 

.046 

Basketball 

42 

F 

-.047 

-.099 

Baseball 

79 

K 

.053 

-.017 

Crew 

61 

Hs 

.077 

-.076 

Fencing 

50 

D 

.049 

.183 

Golf 

28 

Hy 

-.028 

.031 

Gymnastics 

26 

Pd 

.001 

-.069 

Hockey 

28 

Mf 

-.074 

-.076 

Swimming 

51 

Pa 

.189 

.088 

Tennis 

31 

Pt 

.025 

-.188 

Track 

52 

Sc 

-.046 

,088 

Wrestling 

64 

Ma 

-.103 

.053 



Si 

.041 

.016 


Source: W. Morgan and R. W. Johnson. 
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Second discriminant 


• Swimming 


.4 • 6 

Football 


> Gymnastics 
• Crew 


•Golf I •Basketball 


First discriminant 


Figure 11.13 The discriminant means y' = [yi * yi] for each s P ort * 


unit ellipse (circle) centered at the discriminant mean vector y should contain 
approximately a proportion 

P[( Y - Mr)'(Y - my) s 1] *= P[xl * U = -39 
of the points when two discriminants are plotted. ■ 

Using Fisher’s Discriminants to Classify Objects 

Fisher’s discriminants were derived for the purpose of obtaining a low-dimensional 
representation of the data that separates the populations as “ UI J 
though they were derived from considerations of separation, the discriminants also 
provide the basis for a classification rule. We first explain the connection in terms of 
the population discriminants a,-X. 

Setting 


Y k = a*X = Arth discriminant, k ^ s 


(11-64) 


we conclude that 


Y _ ** 

= } has mean vector p iY ~ 


under population n, and covariance matrix I, for all populations. (See Exercise 
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Because the components of Y have unit variances and zero covariances, the 
appropriate measure of squared distance from Y = y to /x iY is 

j 

(y - M;y)’( y - MiY ) = 2 O’/ ~ Pix) 2 

J -1 

A reasonable classification rule is one that assigns y to population 7r k if the square 
of the distance from y to fx kY is smaller than the square of the distance from y to fx lY 
for i ^ k . 

If only r of the discriminants are used for allocation, the rule is 
Allocate x to n k if 

2 (yj - m*v>) 2 = 2 f a y( x - i^k)] 2 
M /-1 

s 2 [ a y(* - Pi)f for all / ^ k (11-65) 

/=i 

Before relating this classification procedure to those of Section 11.5, we look 
more closely at the restriction on the number of discriminants. From Exercise 11.21, 

s = number of discriminants = number of nonzero eigenvalues of X _1 B M 

or ofX~ 1/2 B M X' 1/2 

Now, X _1 B M is p X p, so s ^ p. Further, the g vectors 

Mi - M.M 2 - M.M* - M _ (11-66) 

satisfy (/Lt x — ji) + (/x 2 ~ p) + + (i* g — ji) — gji — gji = 0. That is, the first 

difference pL\ — ji can be written as a linear combination of the last g — 1 differ¬ 
ences. Linear combinations of the g vectors in (11-66) determine a hyperplane of di¬ 
mension q ^ g - 1. Ihking any vector e perpendicular to every /Lt f — ji , and hence 
the hyperplane, gives 

®M e = 2 (Ml - m)(Mi - M)’e = 2 (Mi ~ m) 0 = o 
1=1 1=1 

SO 

X~ 1 B M e = Oe 

There are p — q orthogonal eigenvectors corresponding to the zero eigenvalue. This 
implies that there are q or fewer nonzero eigenvalues. Since it is always true that 
q s g — 1, the number of nonzero eigenvalues s must satisfy s ^ min(p, g — 1). 

Thus, there is no loss of information for discrimination by plotting in two 
dimensions if the following conditions hold. 


Number of Number of Maximum number 

variables populations of discriminants 
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We now present an important relation between the classification rule (11-65) 
and the “normal theory” discriminant scores [see (1149)], 

rf,(x) = m/ 2 -1 * - + ln A 

or, equivalently, 

di(x) - |x'2 _1 x = -j(x - mYl-'i* ~ nd + Inp, 
obtained by adding the same constant -|x'2 _1 x to each d,(x). 

Result 11.6. Let yj - ajx, where a j = £“ 1/2 e ; ande ; is an eigenvector of ST^B^IT 1 / 2 . 
Then 

2 (»' ~ M/v,) 2 = S fa;(x - Hi)] 2 = (x - - Mi) 

j=1 j =1 

= -2d f (x) + x'S^x + 21np,- 

p 

If Ai > • • • > \ s > 0 = A J+1 = • ■ = \ p , 2 ™ M/y ) 2 is constant for all popu- 

'- J+1 ' , 

lations i = 1,2,... so only the first s discriminants y ; , or ^ (>7 Miy ) 2 > con- 

;=i ; 

tribute to the classification. 

Also, if the prior probabilities are such that Pi = P 2 ~ ‘ = P g ~ 1 /g, the rule 
(11-65) with r = s is equivalent to the population version of the minimum TPM 
rule (11-52). 

Proof. The squared distance (x - - /a,-) = (x - /a/) , 2~ 1/2 2~ 1/2 (x - /* f ) 

= (x - /a 2 ) , 2~ 1/2 EE'5) _1/2 (x - /t|), where E = [ei,e 2 ,... ,e p ] is the orthogonal 
matrix whose columns are eigenvectors of (See Exercise 11.21.) 

Since 2T 1/2 e, = a, ora- = e-5T 1/2 , 

~»l(x — Mi) 
e's~ i/2 (x - mi) = a2(x r M,) 

_ a p(x - Mi). 

and 

(x - M.rs-^EE’S-V^x - Ml .) = 2 fa'(x - Mi)] 2 

y=i 

Next, each a y = 2“ 1 / 2 e y , 7 > 5 *, is an (unsealed) eigenvector of X^B^ with eigen¬ 
value zero. As shown in the discussion following (11-66), a } is perpendicular to every 
Mi - m and hence to (/a* - ft) - (/a, - £) = ja* “ Mi to = 1,2,... ,g. The 
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condition 0 a ; '(M* Mi) - M*y, “ Miy, implies that y j - = y ; . - ^ iYj so 

^.2^ ~ ^ /y y) IS constant for all i = 1,2,. ..,g. Therefore, only the first s dis¬ 
criminants y } need to be used for classification. m 

We now state the classification rule based on the first r ^ s sample discriminants. 


Fisher s Classification Procedure Based 
on Sample Discriminants 

Allocate x to n k if 


2 (fy Sty) ~ 2 [**y( x “ x k)] 2 s 2 [**y( x " x/)] 2 for all i * k 
y =1 y =1 


where a, is defined in (11-62), y kj = fijx* and r 


(11-67) 


When the prior probabilities are such that p x = p 2 - • • = p = l/g and r = s 
rule (11-67) is equivalent to rule (11-52), which is based on the*largest linear dis¬ 
criminant score. In addition, if r < s discriminants are used for classification, there 

is a loss of squared distance, or score, of ^ [aj(x - x,-)] 2 for each population tt, 

5 > =r+1 

where ^ f^/( x *i)f is the part useful for classification. 

;=r+l 


Example 11.16 (Classifying a new observation with Fisher's discriminants) Let us 

use the Fisher discriminants 

Pi = ajx = .386*, + .495*2 
= a;x = .938*, - .112x 2 

from Examjjie 11.13 to classify the new observation x' 0 = [1 3] in accordance with 
(11-67). 

Inserting xo = [*oi. * 02 ] = [1 3], we have 

y, = .386*0, + -495*02 = 386(1) + .495(3) = 1.87 
$2 = .938*0, - .112*02 = -938(1) - .112(3) = .60 
Moreover, y kj = a'x*, so that (see Example 11.13) 

hi = aix, = [.386 .495] J = 1.10 
yn = aji, = [-938 -.112] r _ *"j = -1.27 
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Similarly, 

5-21 = ai*2 = 2.37 
5*22 = »2*2 = 

hi = “1*3 = “99 

hi = *2*3 = -22 

Finally, the smallest value of 

2 (h ~ hi? = 2 (* - 

2 =i t-i 

for fc = 1,2,3, must be identified. Using the preceding numbers gives 

2 (h - hi? = (1-87 - 1.10) 2 + (.60 + 1.27) 2 = 4.09 
2-1 

2 (h ~ hi? = (1-87 - 2.37) 2 + (.60 - .49) 2 = .26 
2 = 1 

2 Oj - hi? = (1-87 + -99) 2 + (.60 - .22) 2 = 8.32 , 

2=1 

2 

Since the minimum of ^ (S'/ ~ hj) ocoits when k = 2, we allocate x 0 to 
;=i 

population tt 2 . The situation, in terms of the classifiOers yj, is illustrated schematical¬ 
ly in Figure 11.14. ■ 



Figure I U4 

The points y' __ 

yi = t y 11 »2 ] > y2 = [5*21 > 1 ^ 22 ] 
and y 3 = [y 3 i, ^ 32 ] in the 
classification plane. 
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Comment When two linear discriminant functions are used for classification, 
observations are assigned to populations based on Euclidean distances in the two- 
dimensional discriminant space. 

Up to this point, we have not shown why the first few discriminants are more 
important than the last few. Their relative importance becomes apparent from their 
contribution to a numerical measure of spread of the populations. Consider the sep¬ 
aratory measure 



and (#*i - - jx) is the squared statistical distance from the ith 

population mean /a, to the centroid jt. It can be shown (see Exercise 11.22) that 
A 5 = Ai + A 2 + ■ • * + A p where the Aj 5: A 2 ^ ^ A s are the nonzero eigenvalues 

of IT 1 !* (or 2 ~ 1 / 7 B£ -1/2 ) and \ s+1 ,... , A p are the zero eigenvalues. 


The separation given by A 5 can be reproduced in terms of discriminant means. 
The first discriminant, Y x = eiX" 1/2 X has means fi iY] = and th e squared 

g 

distance ^ (/i iyi - fi Yl ) 2 of the /i/y/s from the central value j± Yl - e[ X“ 1/2 A is Aj. 

i=1 

(See Exercise 11.22.) Since A 5 can also be written as 


- Ai + A 2 + - • + A p 

“ X (M-/y ~ Ay)' (M/y “ Av) 


= X (wv, “ + X (^V 2 " ^Yi) 2 + * * * + X 


(Miy, “ AVp) 


2 


it follows that the first discriminant makes the largest single contribution, Ai, to the 
separative measure A$. In general, the rth discriminant, Y r = e'lT^X, contributes 
A r to Aj. If the next s — r eigenvalues (recall that A 5+1 = A J+2 = * = A p = 0) are 
such that A r+1 + A r+2 + ■ * * + A s is small compared to A x + A 2 + * ♦ • + A r , then the 
last discriminants Y r+: , Y r + 2 , • • *, Y s can be neglected without appreciably decreasing 
the amount of separation. 12 

Not much is known about the efficacy of the allocation rule (11-67). Some insight 
is provided by computer-generated sampling experiments, and Lachenbruch [23] 
summarizes its performance in particular cases The development of the population re¬ 
sult in (11-65) required a common covariance matrix X. If this is essentially true and 
the samples are reasonably large, rule (11-67) should perform fairly well. In any event, 
its performance can be checked by computing estimated error rates. Specifically, 
Lachenbruch’s estimate of the expected actual error rate given by (1T57) should be 
calculated. 


12 See [18] for further optimal dimension-reducing properties. 
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11.7 Logistic Regression and Classification 

Introduction 

The classification functions already discussed are based on quantitative variables. 
Here we discuss an approach to classification where some or all of the variables are 
qualitative. This approach is called logistic regression. In its simplest setting, the 
response variable Y is restricted to two values. For example, Y may be recorded as 
“male” or “female” or “employed” and “not employed.” 

Even though the response may be a two outcome qualitative variable, we can 
always code the two cases as 0 and 1. For instance, we can take male = 0 and 
female = 1. Then the probability p of 1 is a parameter of interest. It represents the. 
proportion in the population who are coded 1. The mean of the distribution of 0’s 
and l’s is also p since 

mean = 0 X (1 ~p) + lXp = p 

The proportion of 0’s is 1 - p which is sometimes denoted as q. 

The variance of the distribution is 

variance = 0 2 x (1 - p) + l 2 X p - p 2 = p(l - p) 

It is clear the variance is not constant. For p = .5, it equals .5 X .5 = .25 while for 
p = .8, it is .8 X .2 = .16. The variance approaches 0 asp approaches either 0 or 1. 

Let the response Y be either 0 or 1. If we were to model the probability of 1 with 
a single predictor linear model, we would write 

P = E(Y | z) = p 0 + PiZ 

and then add an error term e. But there are serious drawbacks to this model. 

• The predicted values of the response Y could become greater than 1 or less than 
0 because the linear expression for its expected value is unbounded. 

• One of the assumptions of a regression analysis is that the variance of Y is con¬ 
stant across all values of the predictor variable Z. We have shown this is not the 
case. Of course, weighted least squares might improve the situation. 

We need another approach to introduce predictor variables or covariates Z into 
the model (sete [26]). Throughout, if the covariates are not fixed by the investigator, 
the approach is to make the models for p(z) conditional on the observed values 
of the covariates Z = z. 

The Logit Model 

Instead of modeling the probability p directly with a linear model, we first consider 
the odds ratio 

odds = ——— 

1 - p 

which is the ratio of the probability of I to the probability of 0. Note, unlike proba¬ 
bility, the odds ratio can be greater than 1. If a proportion .8 of persons will get 
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Figure 11.15 Natural log of 
odds ratio. 


through customs without their luggage being checked, then p = .8 but the odds of 
not getting checked is .8/.2 = 4 or 4 to 1 of not being checked. There is a lack of 
symmetry here since the odds of being checked are 21.8 = 1/4. Taking the natural 
logarithms, we find that ln(4) = 1.386 and ln(l/4) = -1.386 are exact opposites. 

Consider the natural log function of the odds ratio that is displayed in 
Figure 11.15. When the odds x are 1, so outcomes 0 and 1 are equally likely, the nat¬ 
ural log of x is zero. When the odds x are greater than one, the natural log increases 
slowly as x increases. However, when the odds x are less than one, the natural log de¬ 
creases rapidly as x decreases toward zero. 

In logistic regression for a binary variable, we model the natural log of the odds 
ratio, which is called logit(p). Thus 


logit(p) = ln(odds) = In 



(11-69) 


The logit is a function of the probability p. In the simplest model, we assume that the 
logit graphs as a straight line in the predictor variable Z so 


logit(p) = ln(odds) = In 



Po + Pi z 


(11-70) 


In other words, the log odds are linear in the predictor variable. 

Because it is easier for most people to think in terms of probabilities, we can 
convert from the logit or log odds to the probability p. By first exponentiating 


we obtain 



= Po + Pi z 


0(z) 


pjz ) 

1 - p(z) 


exp(/3 0 + Piz) 
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where exp = e = 2.718 is the base of the natural logarithm. Next solving for 0(z)> 
we obtain 


exp(/3 0 + faz) 

1 + exp(/3 0 + p lZ ) 


(11-71) 



which describes a logistic curve. The relation between p and the predictor z is not lin¬ 
ear but has an 5-shaped graph as illustrated in Figure 11.16 for the case p 0 = -1 and 
Pi = 2.The value of /3 0 g ives the value exp(/3 0 )/(l + exp(j3 0 )) for/? when* = 0. 

The parameter pi in the logistic curve determines how quickly p changes with z 
but its interpretation is not as simple as in ordinary linear regression because the re¬ 
lation is not linear, either in z or p Y . However, we can exploit the linear relation for 
log odds. 

To summarize, the logistic curve can be written as 

exp(/3o + Piz) _1_ 

P Z 1 + exp(/3 0 + Piz) 1 P 1 1 + exp(-fio ~ Pi z ) 


Logistic Regression Analysis 


Consider the model with several predictor variables. Let (zp, zp> ■ • •. Zjr) he the val¬ 
ues of the r predictors for the ;-th observation. It is customary, as in normal theory 
linear regression, to set the first entry equal to 1 and ij ~ [1, z ;1 , zp> • • •, Zj r ]'- Con¬ 
ditional on these values, we assume that the observation Y } is Bernoulli with success 


probability p(zj), depending on the values of the covariates.Then 

P(Yj = yj) = ifKzjH i - fo r y, = o> 1 



SO 


E(Yj) = p(xj) and Var^) = p(z;)(l - />(*;)) 
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It is not the mean that follows a linear model but the natural log of the odds ratio. In 
particular, we assume the model 

ln (i~-^p(z)) ~ + ^ lZl + + PrZr ~ P'*j (11-72) 

where P = [P 0 ,Pi, • ■ 

Maximum Likelihood Estimation. Estimates of the p J s can be obtained by the 
method of maximum likelihood. The likelihood L is given by the joint probability 
distribution evaluated at the observed counts y ; . Hence 

L(bo,b u ... y b r ) = 1 - P(*j)) l - y * 

/=! 

TT e y ^ bo+ biZ i l + • • • +brZ 

= n;.id + e i » +6 ^. + - + ^") (11_73) 

The values of the parameters that maximize the likelihood cannot be expressed 
in a nice closed form solution as in the normal theory linear models case. Instead 
they must be determined numerically by starting with an initial guess and iterating 
to the maximum of the likelihood function. Technically, this procedure is called an 
iteratively re-weighted least squares method (see [26]). 

We denote the numerically obtained values of the maximum likelihood esti¬ 
mates by the vector p. 

Confidence Intervals for Parameters. When the sample size is large, p is approxi¬ 
mately normal with mean /J, the prevailing values of the parameters and approxi¬ 
mate covariance matrix 

_ . T 1 

Cov(/J) « 2i>(z,)(l - >(■/))■/■/ (11-74) 

L/-i J 

The square roots of the diagonal elements of this matrix are the large sample esti¬ 
mated standard deviations or standard errors (SE) of the estimators Pq, pi >..., p r 
respectively.The large sample 95% confidence interval for p k is 

p k ± 1.96 SE(p k ) k — 0,1,..., r (11-75) 

The confidence intervals can be used to judge the significance of the individual 
terms in the model for the logit. Large sample confidence intervals for the logit and 
for the population proportion p(z ; ) can be constructed as well. See [17] for details. 

Likelihood Ratio Tests. For the model with r predictor variables plus the constant, 
we denote the maximized likelihood by 

^max — E (Poy /^1> ♦ • ■ » Pr ) 
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If the null hypothesis is H 0 : p k = 0, numerical calculations again give the maximum 
likelihood estimate of the reduced model and, in turn, the maximized value of the 
likelihood 

Lmax, Reduced HP - Pr) 

When doing logistic regression, it is common to test H 0 using minus twice the Jog- 
likelihood ratio 

_ , ( ^max, Reduced \ . „ 

-2 In - - - . (11-76) 

• \ ^max / 

which, in this context, is called the deviance. It is approximately distributed as chi- 
square with 1 degree of freedom when the reduced model has one fewer predictor 
variables. H 0 is rejected for a large value of the deviance. 

An alternative test for the significance of an individual term in the model for the 
logit is due to Wald (see [17]).The Wald test of Ho : P k = 0 uses the test statistic 
Z ~ p k /SE(/3 k ) or its chi-square version Z 2 with 1 degree of freedom. The likeli¬ 
hood ratio test is preferable to the Wald test as the level of this test is typically clos¬ 
er to the nominal a. 

Generally, if the null hypothesis specifies a subset of, say, m parameters are si¬ 
multaneously 0, the deviance is constructed for the implied reduced model and re¬ 
ferred to a chi-squared distribution with m degrees of freedom. 

When working with individual binary observations Tj, the residuals 


Yj ~ p(«y) 


each can assume only two possible values and are not particularly useful. It is better 
if they can be grouped into reasonable sets and a total residual calculated for each 
set. If there are, say, t residuals in each group, sum these residuals and then divide by 
v 7 to help keep the variances compatible. 

We give additional details on logistic regression and model checking following 
and application to classification. 


Classification 

Let the response variable Y be 1 if the observational unit belongs to population 1 
and 0 if it belongs to population 2. (The choice of 1 and 0 for response outcomes is 
arbitrary but convenient. In Example 11.17, we use 1 and 2 as outcomes.) Once a 
logistic regression function has been established, and using training sets for each of 
the two populations, we can proceed to classify. Priors and costs are difficult to 
incorporate into the analysis, so the classification rule becomes 

Assign z to population 1 if the estimated odds ratio is greater 
than 1 or 


p( z) - 

l _ = eXp(A) + 01*1 + • • + PrZr) > 1 
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Equivalently, we have the simple linear discriminant rule 

Assign z to population 1 if the linear discriminant is greater 
than 0 or 

00 + 0!*! +■••+ PrZ r >0 (11-77) 

Example 11.17 (Logistic regression with the salmon data) We introduced the salmon 
data in Example 11.8 (see Table 11.2). In Example 11.8, we ignored the gender of the 
salmon when considering the problem of classifying salmon as Alaskan or Canadian 
based on growth ring measurements. Perhaps better classification is possible if gen¬ 
der is included in the analysis. Panel 11.2 contains the SAS output from a logistic re¬ 
gression analysis of the salmon data. Here the response Y is 1 if Alaskan salmon and 
2 if Canadian salmon. The predictor variables (covariates) are gender (1 if female, 2 if 
male), freshwater growth and marine growth. From the SAS output under Tfesting 
the Global Null Hypothesis, the likelihood ratio test result (see 11-76) with the re¬ 
duced model containing only a p 0 term) is significant at the < .0001 level. At least 
one covariate is required in the linear model for the logit. Examining the significance 
of individual terms under the heading Analysis of Maximum Likelihood Estimates, 
we see that the Wald test suggests gender is not significant (p-value = .7356). On the 
other hand, freshwater growth and marine are significant covariates. Gender can be 
dropped from the model. It is not a useful variable for classification. The logistic re¬ 
gression model can be re-estimated without gender and the resulting function used 
to classify the salmon as Alaskan or Canadian using rule (11-77). 

Thrning to the classification problem, but retaining gender, we assign salmon j 
to population 1, Alaskan, if the linear classifier 

P'z = 3.5054 + .2816 gender + .1264 freshwater + .0486 marine > 0 
The observations that are misclassified are 


Row 

Pop 

Gender 

Freshwater 

Marine 

Linear Classifier 

2 

1 

1 

131 

355 

3.093 

12 

1 

2 

123 

372 

1.537 

13 

1 

1 

123 

372 

1.255 

30 

1 

2 

118 

381 

0.467 

51 

2 

1 

129 

420 

—0.319 

68 

2 

2 

136 

438 

-0.028 

71 

2 

2 

90 

385 

-3.266 


From these misclassifications, the confusion matrix is 


7?!.* Alaskan 

Actual 

TT \: Canadian 


Predicted membership 


ttj : Alaskan tt x : Canadian 
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and the apparent error rate, expressed as a percentage is ~ 

APER ■ sJTlo * 100 ' 7% L 

When performing a logistic classification, it would be preferable to have an estimated 
of the misclassification probabilities using the jackknife (holdout) approach but this . m 
is not currently available in the major statistical software packages. ilfilil 

We could have continued the analysis in Example 11.17 by dropping gender affl§ 
using just the freshwater and marine growth measurements. However, when normal! 
distributions with equal covariance matrices prevail, logistic classification can & 
quite inefficient compared to the normal theory linear classifier (see [7]). 

Logistic Regression with Binomial Responses ^ 

We now consider a slightly more general case where several runs are made at th$ffjj 
same values of the covariates Zj and there are a total of m different sets where these - ^ 
predictor variables are constant. When n } independent trials are conducted withfpfi* 
the predictor variables Zj, the response Yj is modeled as a binomial distribution- 
with probability p{z f ) = P {Success I z y ). 

Because the Y } are assumed to be independent, the likelihood is the product 



L(Po,Pi 


ft (*>'> 


P(*/))^ 


(11-78)- 


where the probabilities p(z ; ) follow the logit model (11-72) 
PANEL I 1.2 SAS ANALYSIS FOR SALMON DATA USING PROC LOGISTIC. 

title ‘Logistic Regression and Discrimination'; 
data salmon; 

infileT11-2.dat'; L PROGF 

input country gender freshwater marine; 
proc logistic desc; 

model country = gender freshwater marine / expb; 


PROGRAM COMMANDS 



Logistic Regression and Discrimination 
The LOGISTIC procedure 


Ordered 

Value 

1 

2 


Model Information 


binary logit 


Response Profile 


country 

2 

1 


Total 

Frequency 

50 

50 


(continues on next page 
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PANEL 11.2 (continued) 


Probability modeled is country = 2. 
Model Fit Statistics 


Criterion 

A1C 

SC 

-2 Log L 


Intercept 

Only 

140.629 
143.235 

138.629 


Intercept 

and 

Covariates 

46.674 
57.094 

38.674 


Test 

Likeljhobd Ratio 
Wald 


Testing Global Null Hypothesis: BETA = 0 

Chi-Square DF Pr > ChiSq 

^ <.booi I 


The LOGISTIC Procedure 
Analysis of Maximum Likelihood Estimates 





|^|||||§ 

jy*' * 


^Ipij^CbiSp 

Exp (Est) 


ILiHE 


0.3006 

0.5835 

33.293 


ispfoast'f'- 

ftsm. 

0.1140 


1.325 

010$ 


fetltllfe 


IS ",, 

1.135 


-0.0486 




- 0.953 


The maximum likelihood estimates p must be obtained numerically because 
there is no closed form expression for their computation. When the total sample size 
is large, the approximate covariance matrix Cov(0) is 


(11-79) 


Cov(0) w 2 n ;P( z /)( 1 ~ P( z i ))*;*/ 
L;=l 


and the i -th diagonal element is an estimate of the variance of It’s square root 
is an estimate of the large sample standard error SE (£ l+ i). 

It can also be shown that a large sample estimate of the variance of the proba¬ 
bility p(zj) is given by 


Vix(p(*k)) « (p(z*)(l “ P(zk)) 2 z/\ - p(Zj))ZjZ/ z k 


Consideration of the interval plus and minus two estimated standard deviations 
from p(zj) may suggest observations that are difficult to classify. 






642 Chapter 11 Discrimination and Classification 


Model Checking. Once any model is fit to the data,it is good practice to investigate 
the adequacy of the fit. The following questions must be addressed. 

• Is there any systematic departure from the fitted logistic model? 

• Are there any observations that are unusual in that they don’t fit the overall 
pattern of the data (outliers)! 

• Are there any observations that lead to important changes in the statistical 
analysis when they are included or excluded (high influence)! 

If there is no parametric structure to the single trial probabilities p(zf) = 
P (Success I z y ), each would be estimated using the observed number of successes 
(l’s) y t in trials. Under this nonparametric model, or saturated model, the contri¬ 
bution to the likelihood for the ;-th case is 

- p(zi)v ry ‘ 

which is maximized by the choices p(zf) = yjlnj for j = 1,2,..., n. Here m = 

The resulting value for minus twice the maximized nonparametric (NP) likelihood 
is 

- 2 In L maxNP = — 2 yjln (^) + (n, - y t ) hi ^1 - ~j + 21n j 

(11-80) 

The last term on the right hand side of (11-80) is common to all models. 

We also define a deviance between the nonparametric model and a fitted model 
having a constant and r—1 predicators as minus twice the log-likelihood ratio or 

G 2 = 22 kln(|) + (nj - yj) (H-81) 

7=1 L \yj/ \nj yj/ J 

where y } = n } p(zf) is the fitted number of successes. This is the specific deviance 
quantity that plays a role similar to that played by the residual (error) sum of 
squares in the linear models setting. 

For large sample sizes, G 2 has approximately a chi square distribution with / 
degrees of freedom equal to the number of observations, m, minus the number of 
parameters p estimated. 

Notice the deviance for the full model, G 2 FulF and the deviance for a reduced 
model, G 2 Reduced , lead to a contribution for the extra predictor terms 

Gl du « d - G 2 Fu „ = -2 in (11-82) 

This difference is approximately x 2 with degrees of freedom df = df Reduced - df Fu n- 
A large value for the difference implies the full model is required. 

When m is large, there are too many probabilities to estimate under the non- 
parametic model and the chi-square approximation cannot be established by exist¬ 
ing methods of proof. It is better to rely on likelihood ratio tests of logistic models 
where a few terms are dropped. 


Logistic Regression and Classification 643 


Residuals andI Goodness-of-Fit Tests. Residuals can be inspected for patterns that 
suggest ack of fit of the logit model form and the choice of predictor variables (co- 
variales). In logistic regression residuals are not as well defined as in the multiple re¬ 
gression models discussed in Chapter 7. Three different definitions of residuals are 


Deviance residuals (df): 


d,m 


~„7-»),n(— 

W 1 ~ P(*j)) J _ 


where the sign of d t is the same as that of y, - nj p{ z f ) and, 

y> ~ °< th e n d i = - Vlrijlln (1 - p(z/))T 
if y, = n p then d, = -V2n y lln p( Zj )) j (U-83) 


Pearson residuals(rf): 


r = yj - HjP (z,) 

- P(Zj)) 


(11-84) 


Standardized Pearson residuals (r SJ ): 


Vi - h . 


(11-85) 


where h Jf is the (;,;)th element in the “hat” matrix H given by equation (11-87). 
Values larger than about 2.5 suggest lack of fit at the particular z . 

An overall test of goodness of fit—preferred especially for smaller sample 
sizes is provided by Pearson’s chi square statistic 



(y, - n ,p(zj )) 2 

n jP(Zj){{ - p{Zj)) 


( 11 - 86 ) 


Notice that the chi square statistic, a single number summary of fit, is the sum of the 
squares of the Pearson residuals. Inspecting the Pearson residuals themselves allows 
us to examine the quality of fit over the entire pattern of covariates 

Another goodness-of-fit test due to Hosmer and Lemeshow [17] is only applic- 
able when the proportion of observations with tied covariate patterns is small and 
all the predictor variables (covariates) are continuous. 


Leverage Points and Influential Observations. The logistic regression equivalent of 
the hat matrix H contains the estimated probabilities p k ( Z] ). The logistic regression 
version of leverages are the diagonal elements h p of this hat matrix. 

H = V~ 1/2 Z(Z'V~ I Z)“ I Z'V~ 1/2 (11-87) 

where V' 1 is the diagonal matrix with (;,;) element n,-p (.z ,)(1 - b(z ,)). V~ m is the 
diagonal matrix with (;,;) element Vn,p(z~)(l - p( Xl j]. 

Besides the leverages given in (11-87), other measures are available. We de¬ 
scribe the most common called the delta beta or deletion displacement. It helps iden¬ 
tify observations that, by themselves, have a strong influence on the regression 
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estimates. This change in regression coefficients, when all observations with the 
same covariate values as the /-th case z } - are deleted, is quantified as 

, rljhjj 

= yzjr ( 11 - 88 ) 

i rijj 

A plot of A/3 j versus j can be inspected for influential cases. 

11.8 Final Comments 

Including Qualitative Variables 

Our discussion in this chapter assumes that the discriminatory or classificatory vari¬ 
ables, X\ , .. •, X p have natural units of measurement. That is, each variable can, 

in principle, assume any real number, and these numbers can be recorded. Often, a 
qualitative or categorical variable may be a useful discriminator (classifier). For ex¬ 
ample, the presence or absence of a characteristic such as the color red may be a 
worthwhile classifier. This situation is frequently handled by creating a variable X 
whose numerical value is 1 if the object possesses the characteristic and zero if the 
object does not possess the characteristic. The variable is then treated like the mea¬ 
sured variables in the usual discrimination and classification procedures. 

Except for logistic classification, there is very little theory available to handle the 
case in which some variables are continuous and some qualitative. Computer simula¬ 
tion experiments (see [22]) indicate that Fisher’s linear discriminant function can per¬ 
form poorly or satisfactorily, depending upon the correlations between the qualitative 
and continuous variables. As Krzanowski [22] notes, “A low correlation in one popu¬ 
lation but a high correlation in the other, or a change in the sign of the correlations be¬ 
tween the two populations could indicate conditions unfavorable to Fisher’s linear 
discriminant function.” This is a troublesome area and one that needs further study. 

Classification Trees 

An approach to classification completely different from the methods discussed in 
the previous sections of this chapter has been developed. (See [5].) It is very com¬ 
puter intensive and its implementation is only now becoming widespread. The new 
approach, called classification and regression trees (CART), is closely related to di¬ 
visive clustering techniques. (See Chapter 12.) 

Initially, all objects are considered as a single group. The group is split into two 
subgroups using, say, high values of a variable for one group and low values for the 
other. The two subgroups are then each split using the values of a second variable. 
The splitting process continues until a suitable stopping point is reached. The values 
of the splitting variables can be ordered or unordered categories. It is this feature 
that makes the CART procedure so general. 

For example, suppose subjects are to be classified as 
7 tj: heart-attack prone 
7 r 2 : not heart-attack prone 

on the basis of age, weight, and exercise activity. In this case, the CART procedure 
can be diagrammed as the tree shown in Figure 11.17. The branches of the tree actually 
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Jt j : Heart-attack prone 

re 2 : Not heart-attack prone Figure 11.17 A classification tree. 

correspond to divisions in the sample space. The region Ri , defined as being over 45, 
being overweight, and undertaking no regular exercise, could be used to classify a 
subject as n ^ heart-attack prone. The CART procedure would try splitting on 
different ages, as well as first splitting on weight or on the amount of exercise. 

The classification tree that results from using the CART methodology with the 
Iris data (see Table 11.5), and variables X 3 - petal length (PetLength) and 
X 4 = petal width (PetWidth), is shown in Figure 11.18. The binary splitting rules are 
indicated in the figure. For example, the first split occurs at petal length = 2.45. 
Flowers with petal lengths < 2.45 form one group (left), and those with petal 
lengths > 2.45 form the other group (right). 


s 2.45 



Figure 11.18 A classification tree 
for the Iris data. 
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The next split occurs with the right-hand side group (petal length > 2.45) at 
petal width = 1.75. Flowers with petal widths ^ 1.75 are put in one group (left), 
and those with petal widths >1.75 form the other group (right). The process con¬ 
tinues until there is no gain with additional splitting. In this case, the process stops 
with four terminal nodes (TN). 

The binary splits form terminal node rectangles (regions) in the positive 
quadrant of the X 3 ,X 4 sample space as shown in Figure 11.19. For example,TN #2 
contains those flowers with 2.45 < petal lengths ^ 4.95 and petal widths < Un¬ 
essentially the Iris Versicolor group. 

Since the majority of the flowers in, for example, TN #3 are species Virginica, a 
new item in this group would be classified as Virginica. That is,TN #3 and TN #4 are 
both assigned to the Virginica population. We see that CART has correctly classified 
50 of 50 of the Setosa flowers, 47 of 50 of the Versicolor flowers, and 49 of 50 of the 

4 

Virginica flowers. The APER = = -027. This result is comparable to the result 

obtained for the linear discriminant analysis using variables X 3 and X 4 discussed in 
Example 11.12. 

The CART methodology is not tied to an underlying population probability 
distribution of characteristics. Nor is it tied to a particular optimality criterion. In 
practice, the procedure requires hundreds of objects and, often, many variables. 
The resulting tree is very complicated. Subjective judgments must be used to 
prune the tree so that it ends with groups of several objects rather than all 
single objects. Each terminal group is then assigned to the population holding the ma¬ 
jority membership. A new object can then be classified according to its ultimate group. 

Breiman, Friedman, Olshen, and Stone [5] have developed special-purpose 
software for implementing a CART analysis. Also, Loh (see [21] anc| [25]) has de¬ 
veloped improved classification tree software called QUEST 13 and CRUISE. 14 
Their programs use several intelligent rules for splitting and usually produces a 
tree that often separates groups well. CART has been very successful in data min¬ 
ing applications (see Supplement 12A). 

7- o 0 0° * 1 Setosa 

0 O° q a -*■ 2 Versicolar 
6 1 TM^ n ° i i8°0° Q 3 Virginica 



0.0 0.5 1.0 1.5 2.0 2.5 

PetWidth 


Figure 11.19 Classification tree terminal nodes (regions) in the petal width, petal 
length sample space. 

13 Available for download at www.stat.wisc.eduMohyquest.html 
14 Available for download at www.stat.wisc.edu/~loh/cruise.html 
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Neural Networks 


A neural network (NN) is a computer-intensive, algorithmic procedure for 
transforming inputs into desired outputs using highly connected networks of 
relatively simple processing units (neurons or nodes). Neural networks are modeled 
after the neural activity in the human brain.The three essential features, then, of an 
NN are the basic computing units (neurons or nodes), the network architecture 
describing the connections between the computing units, and the training 
algorithm used to find values of the network parameters (weights) for performing a 
particular task. 

The computing units are connected to one another in the sense that the out¬ 
put from one unit can serve as part of the input to another unit. Each computing 
unit transforms an input to an output using some prespecified function that is 
typically monotone, but otherwise arbitrary. This function depends on constants 
(parameters) whose values must be determined with a training set of inputs and 
outputs. 

Network architecture is the organization of computing units and the types of 
connections permitted. In statistical applications, the computing units are arranged 
in a series of layers with connections between nodes in different layers, but not be¬ 
tween nodes in the same layer. The layer receiving the initial inputs is called the 
input layer. The final layer is called the output layer. Any layers between the input 
and output layers are called hidden layers. A simple schematic representation of a 
multilayer NN is shown in Figure 11.20. 


Output 


Middle (hidden) 



Figure 11.20 A neural network with one hidden layer. 
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Neural networks can be used for discrimination and classification. When they 
are so used, the input variables are the measured group characteristics X u 
X 2 ,..., X p * and the output variables are categorical variables indicating group 
membership. Current practical experience indicates that properly constructed neur¬ 
al networks perform about as well as logistic regression and the discriminant func¬ 
tions we have discussed in this chapter. Reference [30] contains a good discussion of 
the use of neural networks in applied statistics. 

Selection of Variables 

In some applications of discriminant analysis, data are available on a large number 
of variables. Mucciardi and Gose [27] discuss a discriminant analysis based on 157 
variables. 15 In this case, it would obviously be desirable to select a relatively small 
subset of variables that would contain almost as much information as the original 
collection.This is the objective of stepwise discriminant analysis , and several popular 
commercial computer programs have such a capability. 

If a stepwise discriminant analysis (or any variable selection method) is 
employed, the results should be interpreted with caution. (See [28].) There is no 
guarantee that the subset selected is “best,” regardless of the criterion used to make 
the selection. For example, subsets selected on the basis of minimizing the apparent 
error rate or maximizing “discriminatory power” may perform poorly in future 
samples. Problems associated with variable-selection procedures are magnified if 
there are large correlations among the variables or between linear combinations of 
the variables. 

Choosing a subset of variables that seems to be optimal for a given data set is 
especially disturbing if classification is the objective. At the very least, the derived 
classification function should be evaluated with a validation sample. As Murray [28] 
suggests, a better idea might be to split the sample into a number of batches and 
determine the “best” subset for each batch. The number of times a given variable 
appears in the best subsets provides a measure of the worth of that variable for 
future classification. 


Graphics 

Sophisticated computer graphics now allow one visually to examine multivariate 
data in two and three dimensions. Thus, groupings in the variable space for any 
choice of two or three variables can often be discerned by eye. In this way, poten¬ 
tially important classifying variables are often identified and outlying, or “atypical,” 
observations revealed. Visual displays are important aids in discrimination and clas¬ 
sification, and their use is likely to increase as the hardware and associated comput¬ 
er programs become readily available. Frequently, as much can be learned from a 
visual examination as by a complex numerical analysis. 

Practical Considerations Regarding Multivariate Normality 

The interplay between the choice of tentative assumptions and the form of the re¬ 
sulting classifier is important. Consider Figure 11.21, which shows the kidney¬ 
shaped density contours from two very nonnormal densities. In this case, the normal 
theory linear (or even quadratic) classification rule will be inadequate compared to 
another choice. That is, linear discrimination here is inappropriate. 

Often discrimination is attempted with a large number of variables, some of 
which are of the presence-absence, or 0-1, type. In these situations and in others 
with restricted ranges for the variables, multivariate normality may not be a sensible 
assumption. As we have seen, classification based on Fisher’s linear discriminants 
can be optimal from a minimum ECM or minimum TPM point of view only when 
multivariate normality holds. How are we to interpret these quantities when nor¬ 
mality is clearly not viable? 

In the absence of multivariate normality, Fisher’s linear discriminants can be 
viewed as providing an approximation to the total sample information. The values 
of the first few discriminants themselves can be checked for normality and rule 
(11-67) employed. Since the discriminants are linear combinations of a large num¬ 
ber of variables, they will often be nearly normal. Of course, one must keep in mind 
that the first few discriminants are an incomplete summary of the original sample in¬ 
formation. Classification rules based on this restricted set may perform poorly, while 
optimal rules derived from all of the sample information may perform well. 


Testing for Group Differences 

We have pointed out, in connection with two group classification, that effective allo¬ 
cation is probably not possible unless the populations are well separated. The same 
is true for the many group situation. Classification is ordinarily not attempted, un¬ 
less the population mean vectors differ significantly from one another. Assuming 
that the data are nearly multivariate normal, with a common covariance matrix, 
MANOVA can be performed to test for differences in the population mean vectors. 
Although apparent significant differences do not automatically imply effective clas¬ 
sification, testing is a necessary first step. If no significant differences are found,con¬ 
structing classification rules will probably be a waste of time. 




15 Imagine the problems of verifying the assumption of 157-variate normality and simultaneously 
estimating, for example, the 12,403 parameters of the 157 x 157 presumed common covariance matrix! 


Figure 11.21 Two nonnormal 
populations for which linear 
discrimination is inappropriate. 
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exercises 


Consider the two data sets fla 

'3 7] [“6 9~j -1 

X, = 2 4 and X 2 = 5 7 

_4 7 J [4 8 J J3S 

for which 

" 3 I M 2 

Xl = kr X2 = W S 

and 

[ill 

a P° 0,cd [ 1 2 J 

(a) Calculate the linear discriminant function in (11-19). ? 

(b) Classify the observation xq = [2 7] as population or population 7 r 2 , using RuleAfl 

(11-18) with equal priors and equal costs. 

(a) Develop a linear classification function for the data in Example 11.1 using (11-19). 

(b) Using the function in (a) and (11-20), construct the “confusion matrix” by classifying 
the given observations. Compare your classification results with those of Figure 11.1, 
where the classification regions were determined “by eye.” (See Example 11.6.) 

(c) Given the results in (b), calculate the apparent error rate (APER). 

(d) State any assumptions you make to justify the use of the method in Parts a and b. 

Prove Result 11.1. SiS 

Hint: Substituting the integral expressions for P( 2II) and P(112) given by (11-1) and . " 
(11-2), respectively,into (11-5) yields f|j| 

ECM = c(2 ! \)pi f fi(x)dx + c(l \2)p 2 [ f 2 {x)dx 
J /?2 J 

Noting that H = R x U R 2 , so that the total probability 

1 = //l(*)d*= / /i(x)dx + / 

Ja Jr j Jr 2 

we can write 

ECM = c(2ll)/?i[l - jf /,(x)dxj + c(l\2)p 2 J" f 2 (x)dx 


we can write 


By the additive property of integrals (volumes), 


ECM = 


[c(l I 2)p 2 f 2 (x) - c(2\l)p l f l (x)]dx + c(2ll)pi 


Now, pi , P 2 , c( 112), and c(2 11) are nonnegative. In addition,/i(x) and / 2 (x) are non 
negative for all x and are the only quantities in ECM that depend on x. Thus, ECM. 
minimized if R x includes those values x for which the integrand 

[c(l I 2)/7 2 / 2 (x) - c(2ll)pi/ 1 (x)] <0 | 

and excludes those x for which this quantity is positive. 
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I 1.4. A researcher wants to determine a procedure for discriminating between two multivari¬ 
ate populations. The researcher has enough data available to estimate the density 
functions /i(x) and / 2 (x) associated with populations 7 r x and 7 t 2 , respectively. Let 
c(2ll) = 50 (this is the cost of assigning items as 7 r 2 , given that 7 t y is true) and 
c(l12) = 100. 

In addition, it is known that about 20% of all possible items (for which the 
measurements x can be recorded) belong to 7 r 2 . 

(a) Give the minimum ECM rule (in general form) for assigning a new item to one of 
the two populations. 

(b) Measurements recorded on a new item yield the density values / x (x) — .3 and 
/ 2 (x) = .5. Given the preceding information, assign this item to population 7ri or 
population 7 r 2 . 

11.5. Show that 

-j(x - Mi)'X _1 (x - Mi) + H x - M2)'X _1 (x - M 2 ) 

= (Mi “ M2)'2 T'x - |(mi ~ M 2 )'X-'(Mi + M 2 ) 

[see Equation (11-13).] 

11.6. Consider the linear function Y — a'X.Let£(X) = /ijandCov(X) = X if X belongs 
to population ttj . Let £(X) = /* 2 and Cov(X) = X if X belongs to population 7r 2 . Let 
m = \(&\y + M 2 Y) = |(a'/*i + a'/t 2 ). Given that a' = (/*i - /^J'X** 1 , show each 
of the following. 

(a) Efa'XM - m = a'/i 1 - m > 0 

(b) £(a'X|7r 2 ) - m = a'/t 2 - m < 0 

Hint: Recall that X is of full rank and is positive definite, so X -1 exists and is positive 
definite. 

11.7. Let f\{x) = (1 - | x |) for | x | ^ 1 and / 2 (jc) = (1 - | x - .5 |) for -.5 ^ x ^ 1.5. 

(a) Sketch the two densities. 

(b) Identify the classification regions when p x - p 2 and c(l 12) = c(211). 

(c) Identify the classification regions when p { = .2 and c(l I 2) = c(2l 1). 

11.8. Refer to Exercise 11.7. Let f\(x) be the same as in that exercise, but take 
f 2 (x) = 1(2 - |jc - .5|) for -1.5 £ * s 2.5. 

(a) Sketch the two densities. 

(b) Determine the classification regions when p x = p 2 and c(l!2) = c(2ll). 

11.9. For g = 2 groups, show that the ratio in (11-59) is proportional to the ratio 

/ squared distance \ 

^between means of Y J (Miy — M 2 y) 2 (a'/ii — a'/i 2 ) 

(variance of Y) c j\ aXa 

= ~ M2XM1 ~ M 2 )'» = (a ^) 2 

a'Xa a'Xa 

where 6 = (/*i - /t 2 ) is the difference in mean vectors. This ratio is the population 
counterpart of (11-23). Show that the ratio is maximized by the linear combination 

a = cX -1 S = cX -1 (Mi “ M 2 ) 


for any c * 0. 
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Hint: Note that (/a, - jx) (fx, ~ fx)' = |(mi ~ M 2 )(Mi ~ M 2 )' for i = 1,2, where 

fX = |(/*1 + fX 2 ). 

I 1.10. Suppose that nj = 11 and w 2 ~ 12 observations are made on two random variables X l 
and X 2 > where X } and X 2 are assumed to have a bivariate normal distribution with a 
common covariance matrix X, but possibly different mean vectors /x } and fx 2 for the two 
samples. The sample mean vectors and pooled covariance matrix are 



(a) Test for the difference in population mean vectors using Hotelling’s two-sample 
r 2 -statistic. Let a = .10. 

(b) Construct Fisher’s (sample) linear discriminant function. [See (11-19) and (11-25).] 

(c) Assign the observation xq = [0 1] to either population tt y or 77 2 . Assume equal 

costs and equal prior probabilities. 

11.1 I. Suppose a univariate random variable A" has a normal distribution with variance 4. If A" 
is from population 77 ] , its mean is 10; if it is from population 7 t 2 , its mean is 14. Assume 
equal prior probabilities for the events Al = X is from population 7 r 1 and A2 — X is 
from population 7 t 2 , and assume that the misclassification costs c (2 II) and c(l 12 ) are 
equal (for instance, $10). We decide that we shall allocate (classify) X to population 77 ] if 
X < c, for some c to be determined, and to population 77 2 if X > c. Let Bl be the 
event X is classified into population 77 z and B2 be the event X is classified into popula¬ 
tion 7 r 2 . Make a table showing the following: P(Bl\A2), P{B2\A\), P(A1 and B2), 
P(A2 and Bl), P(misclassification), and expected cost for various values of c. For what 
choice of c is expected cost minimized? The table should take the following form: 


c 

P(B1 \A2) 

P(B21A1) 

P(A1 and B2) 

P(A2and Bl) 

F(error) 

Expected 

cost 

10 

14 



! 





What is the value of the minimum expected cost? 

I 1.12. Repeat Exercise 11.11 if the prior probabilities of Al and A2 are equal, but 
c(2II) = $5 and c(l |2) = $15. 

11.13. Repeat Exercise 11.11 if the prior probabilities of Al and A2 are P(A 1) = .25 and 
P(A2) = .75 and the misclassification costs are as in Exercise 11.12. 

11.14. Consider the discriminant functions derived in Example 11.3. Normalize a using (1T21) 
and (11-22). Compute the two midpoints m\ and m 2 corresponding to the two choices of 
normalized vectors, say, a* and a 2 . Classify Xq = [-.210, -.044] with the function 
y* = a*'x 0 f° r the two cases. Are the results consistent with the classification obtained 
for the case of equal prior probabilities in Example 11.3? Should they be? 

11.15. Derive the expressions in (11-27) from (11-6) when /i(x) and / 2 (x) a re multivariate 
normal densities with means /x 1 ,jx 2 and covariances Xi, X 2 , respectively. 
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I LI6. Suppose x comes from one of two populations: 

ttj: Normal with mean /x j and covariance matrix X] 

7 r 2 : Normal with mean /x 2 and covariance matrix X 2 

If the respective density functions are denoted by /i(x) and / 2 (x), find the expression 
for the quadratic discriminator 


Q ~ In 



If Xi = X 2 = X, for instance, verify that Q becomes 

- M2)’2 _1 X “ 2(Ml - + M2) 


11.17. Suppose populations 7 T] and 7r 2 are as follows: 



Population 

77 ] 

7 T 2 

Distribution 

Normal 

Normal 

Mean /i 

[10,15]' 

[10,25]' 

Covariance X 

[[« “1 [ 

20 - 7 " 

|_12 32 J [ 

-7 5_ 



Assume equal prior probabilities and misclassifications costs of c(2!l) - $10 and 
c(l 12) = $73.89. Find the posterior probabilities of populations 77-2 and 7t 2 , P{tt\ lx) 
and F( 7 r 2 lx), the value of the quadratic discriminator Q in Exercise 11.16, and the 
classification for each value of x in the following table: 


X 

n* ii*) 


Q 

Classification 

[10,15]' 
[12,17]' 

[30,35]' 






{Note: Use an increment of 2 in each coordinate—11 points in all.) 


Show each of the following on a graph of the x x , x 2 plane. 

(a) The mean of each population 

(b) The ellipse of minimal area with probability .95 of containing x for each population 

(c) The region (for population 7T|) and the region O-/?! = R 2 (for population 7 t 2 ) 

(d) The 11 points classified in the table 

11.18. If B is defined as c(/xj — fx 2 )(/xi ~ /x 2 ) r for some constant c, verify that 
e = cX -1 (/u! - /x 2 ) is in fact an (unsealed) eigenvector of X -1 B, where X is a covari¬ 
ance matrix. 

11.19. (a) Using the original data sets X! and X 2 given in Example 11.7, calculate x it S f , 

i = 1,2, and S p00led , verifying the results provided for these quantities in the 
example. 
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(b) Using the calculations in Part a, compute Fisher’s linear discriminant function and 
use it to classify the sample observations according to Rule (11-25). Verify that the 
confusion matrix given in Example 11.7 is correct. 

(c) Classify the sample observations on the basis of smallest squared distance D*(x) of 
the observations from the group means xj and x 2 . [See (11-54).] Compare the re-- 
suits with those in Part b. Comment. 

11.20. The matrix identity (see Bartlett [3]) 


c-l 

3 W. pooled 


x k) Spooled ( x // *k) 


*k) (*H ~ X k )'S^ led 


* (n k — l)(n - 2) 

allows the calculation of SJ/ poo | ed from Sp^,i cd . Verify this identity using the data from 
Example 11.7. Specifically, set n = n x + « 2 , k - 1, and x' H = [2,12]. Calculate 
S// 1 ,pooled usin S the fuI1 data Spooled and x,, and compare the result with S^ pooled in 


Example 11.7. 
11.21. Let A, > A 2 > 


A, > 0 denote the s ^ min(g - l,p) nonzero eigenvalues of 


X l B M and e ] ,e 2 ,...,e J the corresponding eigenvectors (scaled so that e'Xe 
Show that the vector of coefficients a that maximizes the ratio 


r * 

la a ' 2 (#*f ~ /*)(/». - /*)' 
_ Li=i 


is given by aj = . The linear combination aiX is called the first discriminant . Show 

that the value a 2 — e 2 maximizes the ratio subject to Cov(aiX,a 2 X) = 0. The linear 
combination a 2 X is called the second discriminant. Continuing, a A . = e k maximizes the 
ratio subject to 0 = Cov (a*X, a}X), / < k , and a*X is called the A:th discriminant. 
Also, Var(a;X) = 1,/ = 1 , [See (11-62) for the sample equivalent.] 

Hint: We first convert the maximization problem to one already solved. By the spectral 
decomposition in (2-20), X = P'AP where A is a diagonal matrix with positive 
elements A,. Let A* /2 denote the diagonal matrix with elements VaB y (2-22), the 
symmetric square-root matrix X 1/7 = P'A t/2 P and its inverse X -1 '" 2 = P'A -1/2 P satisfy 
X 1/2 X I/2 = X, X 1/2 X~ 1/2 =1 = X“ ,/2 X ,/2 and X“ 1/2 X _]/2 = X' 1 . Next, set 

u = X 1/2 a 

so u'u-a'X 1/2 X 1/2 a =a'Xa and u'X~ 1/2 B M X _,/2 u=a , X ,/2 X“ l/2 B. M X~ ]/2 X 1/2 a s: = a'B^a. 
Consequently, the problem reduces to maximizing 

u'X- ,/2 B m X- , ' 2 u 


over u. From (2-51), the maximum of this ratio is Aj, the largest eigenvalue of 
X“ 1/2 B m X _ 1 ^ 2 . This maximum occurs when u = e 1} the normalized eigenvecto r^ 
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associated with A,. Because e 2 = u = X ,/2 a], or = X“ 1/2 ex, Var(a / ,X) = a'.Xa, = 

e'jZ'^ZZ ,/2 e, = e' 1 S _I/2 Z l/2 Z 1/2 Z _I/2 e 1 = eje, = 1. By (2-52), u 1 e, maximizes the 
preceding ratio when u - e 2 , the normalized eigenvector corresponding to A 2 . Fbr this 
choice, a 2 = X~y 2 e 2 , and Cov(a^X,aiX) = a^Za, = e^^Z^e, = e$e, = 0, 
since e 2 1 e,. Similarly, Var(a 2 X)= a 2 Za 2 = e 2 e 2 = 1. Continue in this fashion for 
Note that if A and e are an eigenvalue-eigenvector pair 

Z-i/2 B>< z-i/ 2 e = Ae 

and multiplication on the left by Z“ 1/2 gives 

x -i/ 2 j-i/ 2 B ^ x -i/ 2 e = AZ“ 1/2 e or Z' 1 B M (Z" l/2 e) = A(Z" ,/2 e) 

Thus, Z" 1 B„ has the same eigenvalues as Z^B^Z’ 1 / 2 , but the corresponding eigenvec- 
tor is proportional to X l/2 e = a, as asserted. 

11.22. Show that Af = A, + A 2 + • • • + A„ = A, + A 2 + - • + A t , where A,, A 2 ,..., A* are the 
nonzero eigenvalues ofZ-'B„ (or Z^B^'/ 2 ) and is given by (11-68). Also, show 
that A, + A 2 + ••• + A, is the resulting separation when only the first r discriminants, 
Y\ , .r, are used. 

Hint: Let P be the orthogonal matrix whose /th row e- is the eigenvector of X _,/2 B X _1/2 
corresponding to the / th largest eigenvalue, / = 1,2,...,/?. Consider ^ 

T eiX~ 1/2 X"|* 

= Y ? = e;x- /2 x =pz-/ 2 x 

- K pJ Le;Z -1/2 X_ 

Now ,My = £(Ylw,) = PZ- 1/2 M,and/Z K = PZ' 1/2 /l,so 

(M/y - My)'(M,y - My) = (ft, - jx )'Z“ 1/2 P'PZ~ 1/2 (Mi ~ /Z) 

= (m; - m)'Z''(m/ ~ M) 

8 

Therefore, ^ (/x /y - My)'(/*,y - ji Y ). Using Kj, we have 

/=i 

* , g 

2 (M,y, - My,) = £ *1^ 1/2 (M f - M)(M, - M)'X- 1/2 e, 

I~1 1 =] 

= eiZ-’^Z-^e, = A, 
because e, has eigenvalue A,. Similarly, V 2 produces 

2 (M«y 2 ~ My 2 ) 2 = eJZ“ ,/2 B /1 Z~ 1/2 e 2 = A 2 
1 = 1 


2 (M/y p ~ Hr/ = e^E-'^Z-^e, = A p 


and Y p produces 
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Thus, 

g 

A| = 2 (PiY ~ PvY(PiY - Py) 

/= 1 

S J ^ 2 

= 2 (MiTi “ PY X ) 2 + 2 (MiT 2 "* PY 2 ) 2 + *'* + S “ *%) 

/=1 »=1 ,=1 

= Ai + A 2 + • • * + = Ai + A 2 + • * • + Ay 

since A, + 1 = = \ p = 0. If only the first r discriminants are used, their contribution to 

Aj is Ai + A 2 + ■ * ■ + A r . 

The follow ing exercises require the use of a computer. 

I 1.23. Consider the data given in Exercise 1.14. 

(a) Check the marginal distributions of the jr,’s in both the multiple-sclerosis (MS) 
group and non-multiple-sclerosis (NMS) group for normality by graphing the 
corresponding observations as normal probability plots. Suggest appropriate data 
transformations if the normality assumption is suspect. 

(b) Assume that % x = X 2 = X. Construct Fisher’s linear discriminant function. Do all 
the variables in the discriminant function appear to be important? Discuss your 
answer. Develop a classification rule assuming equal prior probabilities and equal 
costs of misclassification. 

(c) Using the results in (b), calculate the apparent error rate. If computing resources 
allow, calculate an estimate of the expected actual error rate using Lachenbruch’s 
holdout procedure. Compare the two error rates. 

11.24. Annual financial data are collected for bankrupt firms approximately 2 years prior to their 
bankruptcy and for financially sound firms at about the same time. The data on four vari¬ 
ables, X x = CF/TD - (cash flow)/(total debt), = NI/TA = (net income)/(total as¬ 
sets),^ = CA/CL = (current assets)/(current liabilities), and X 4 - CA/NS - (current 
assets)/(net sales), are given in Table 11.4. 

(a) Using a different symbol for each group, plot the data for the pairs of observations 
(x ly x 2 ), (xi,xi) and (^ 1 ,^ 4 ). Does it appear as if the data are approximately 
bivariate normal for any of these pairs of variables? 

(b) Using the n { — 21 pairs of observations (*!, x 2 ) for bankrupt firms and the n 2 = 25 
pairs of observations (x u x 2 ) for nonbankrupt firms, calculate the sample mean vec¬ 
tors X! and x 2 and the sample covariance matrices Si and S 2 . 

(c) Using the results in (b) and assuming that both random samples are from bivariate 
normal populations, construct the classification rule (11-29) with p l = p 2 and 
c(l 12 ) = c( 211 ). 

(d) Evaluate the performance of the classification rule developed in (c) by computing 
the apparent error rate (APER) from (11-34) and the estimated expected actual 
error rate E (AER) from (11-36). 

(e) Repeat Parts c and d, assuming that p x = .05, p 2 = .95, and c(l 12) = c(211). Is 
this choice of prior probabilities reasonable? Explain. 

(f) Using the results in (b), form the pooled covariance matrix S p 00 i ed , and construct 
Fisher’s sample linear discriminant function in (11-19). Use this function to classify 
the sample observations and evaluate the APER. Is Fisher’s linear discriminant 
function a sensible choice for a classifier in this case? Explain. 

(g) Repeat Parts b-e using the observation pairs (jc 1 ( x 3 ) and Do some van_ 

ables appear to be better classifiers than others? Explain. 

(h) Repeat Parts b-e using observations on all four variables (X x , X 2 , X 3 , X 4 ). 
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| 1.25. The annual financial data listed in Table 11.4 have been analyzed by Johnson [19] with a 
view toward detecting influential observations in a discriminant analysis. Consider vari¬ 
ables Xi = CF/TD and X 3 = CA/CL. 

(a) Using the data on variables X 1 and X 3% construct Fisher’s linear discriminant func¬ 
tion. Use this function to classify the sample observations and evaluate the APER. 
[See (11-25) and (11-34).] Plot the data and the discriminant line in the (x,, x 3 ) co¬ 
ordinate system. 

(b) Johnson [19] has argued that the multivariate observations in rows 16 for bankrupt 
firms and 13 for sound firms are influential. Using the X { , X 3 data, calculate Fisher’s 
linear discriminant function with only data point 16 for bankrupt firms deleted. Re¬ 
peat this procedure with only data point 13 for sound firms deleted. Plot the respec¬ 
tive discriminant lines on the scatter in part a, and calculate the APERs, ignoring the 
deleted point in each case. Does deleting either of these multivariate observations 
make a difference? (Note that neither of the potentially influential data points is 
particularly “distant” from the center of its respective scatter.) 

I 1,26. Using the data in Table 11.4, define a binary response variable Z that assumes the value 
0 if a firm is bankrupt and 1 if a firm is not bankrupt. Let X - CA/CL, and consider the 
straight-line regression of Z on Z. 

(a) Although a binary response variable does not meet the standard regression assump¬ 
tions, consider using least squares to determine the fitted straight line for the X, Z 
data. Plot the fitted values for bankrupt firms as a dot diagram on the interval [0,1]. 
Repeat this procedure for nonbankrupt firms and overlay the two dot diagrams. A 
reasonable discrimination rule is to predict that a firm will go bankrupt if its fitted 
value is closer to 0 than to l.That is, the fitted value is less than .5. Similarly, a firm is 
predicted to be sound if its fitted value is greater than .5. Use this decision rule to 
classify the sample firms. Calculate the APER. 

(b) Repeat the analysis in Part a using all four variables, X^ ,.. •, X 4 . Is there any change 
in the APER? Do data points 16 for bankrupt firms and 13 for nonbankrupt firms 
stand out as influential? 

(c) Perform a logistic regression using all four variables. 

11.27. The data in Table 11.5 contain observations on X 2 = sepal width and X 4 = petal width 
for samples from three species of iris.There are n, = n 2 = «3 = 50 observations in each 


sample. 

(a) Plot the data in the ( Jt 2 , jc 4 ) variable space. Do the observations for the three groups 
appear to be bivariate normal? 
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(b) Assume that the samples are from bivariate normal populations with a common 
co variance matrix. Test the hypothesis H 0 : p , = p 2 = M 3 versus//f. at least.one /t, 
is different from the others at the a = .05 significance level. Is the assumption of a 
common covariance matrix reasonable in this case? Explain. 

(c) Assuming that the populations are bivariate normal, construct the quadratic 

discriminate scores dp(x) given by (11-47) with p 1 = p 2 = Pz = b Using Rule 
(11-48), classify the new observation x f 0 = [3.5 1.75] into population tt u tt 2 , or 

3 - 

(d) Assume that the covariance matrices X, are the same for all three bivariate normal 

populations. Construct the linear discriminate score d f (x) given by (11-51), and use 
it to assign Xq = [3.5 1.75] to one of the populations 7r,, i = 1,2,3 according to 

(11-52). Take px ~ Pi ~ Pi ~ 3 * Compare the results in Parts c and d. Which 
approach do you prefer? Explain. 

(e) Assuming equal covariance matrices and bivariate normal populations, and suppos¬ 
ing that p x = p 2 = P 3 = b allocate x o = I 3 - 5 1J5 ] t0 or tt 3 using Rule 

(11-56). Compare the result with that in Part d. Delineate the classification regions 

R 2 , and R 3 on your graph from Part a determined by the linear functions 

^i(x 0 ) in (11-56). , . 

(f) Using the linear discriminant scores from Part d, classify the sample observations. 
Calculate the APER and £(AER). (To calculate the latter, you should use Lachen- 
bruch’s holdout procedure. [See (11-57).]) 

11.28. Darroch and Mosimann [ 6 ] have argued that the three species of iris indicated in 

Table 11.5 can be discriminated on the basis of “shape” or scale-free information alone. 

Let Y\ = X\ /X 2 be sepal shape and Y 2 = X 3 /X 4 be petal shape. 

(a) Plot the data in the (log T,, log Y 2 ) variable space. Do the observations for the three 
groups appear to be bivariate normal? 

(b) Assuming equal covariance matrices and bivariate normal populations, and 
supposing that px = p 2 = Pi = b construct the linear discriminant scores d f (x) 
given by <11-51) using both variables Jog Yj, log Y 2 and each variable individually. 
Calculate the APERs. 

(c) Using the linear discriminant functions from Part b, calculate the holdout estimates 
of the expected AERs, and fill in the following summary table: 


Variable(s) 

Misclassification rate 

logY, 

logY 2 

log Y), log Y 2 



Compare the preceding misclassification rates with those in the summary tables in 
Example 31.12. Does it appear as if information on shape alone is an effective dis¬ 
criminator for these species of iris? 

(d) Compare the corresponding error rates in Parts b and c. Given the scatter plot in 
Part a, would you expect these rates to differ much? Explain. 

11.29. The GPA and GMAT data alluded to in Example 11.11 are listed in Table 11.6. 

(a) Using these data, calculate x,,x 2 , x 3 ,x, and S pooled and thus verify the results for 
these quantities given in Example 11.11. 
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Table 11.6 Admission Data for Graduate School of Business 


7 r\: Admit 

7T 2 

Do not admit 

7t 3 : Borderline 

Applicant 

no. 

GPA 

(*i) 

GMAT 

(*2) 

Applicant 

no. 

GPA 

(* 1 ) 

GMAT 

(* 2 ) 

Applicant 

no. 

GPA 

(* 1 ) 

GMAT 

(* 2 ) 

1 

2.96 

596 

32 

2.54 

446 

60 

2.86 

494 

2 

3.14 

473 

33 

2.43 

425 

61 

2.85 

496 

3 

3.22 

482 

34 

2.20 

474 

62 

3.14 

419 

4 

3.29 

527 

35 

2.36 

531 

63 

3.28 

371 

5 

3.69 

505 

36 

2.57 

542 

64 

2.89 

447 

6 

3.46 

693 

37 

2.35 

406 

65 

3.15 

313 

7 

3.03 

626 

38 

2.51 

412 

66 

3.50 

402 

8 

3.19 

663 

39 

2.51 

458 

67 

2.89 

485 

9 

3.63 

447 

40 

2.36 

399 

68 

2.80 

444 

10 

3.59 

588 

41 

2.36 

482 

69 

3.13 

416 

11 

3.30 

563 

42 

2.66 

420 

70 

3.01 

471 

12 

3.40 

553 

43 

2.68 

414 

71 

2.79 

490 

13 

3.50 

572 

44 

2.48 

533 

72 

2.89 

431 

14 

3.78 

591 

45 

2.46 

509 

73 

2.91 

446 

15 

3,44 

692 

46 

2.63 

504 

74 

2.75 

546 

16 

3.48 

528 

47 

2.44 

336 

75 

2.73 

467 

17 

3.47 

552 

48 

2.13 

408 

76 

3.12 

463 

18 

3.35 

520 

49 

2,41 

469 

77 

3.08 

440 

19 

3.39 

543 

50 

2.55 

538 

78 

3.03 

419 

20 

3.28 

523 

51 

2.31 

505 

79 

3.00 

509 

21 

3.21 

530 

52 

2.41 

489 

80 

3.03 

438 

22 

3.58 

564 

53 

2.19 

411 

81 

3.05 

399 

23 

3.33 

565 

54 

2.35 

321 

82 

2.85 

483 

24 

3.40 

431 

55 

2.60 

394 

83 

3.01 

453 

25 

3.38 

605 

56 

2.55 

528 

84 

3.03 

414 

26 

3.26 

664 

57 

2.72 

399 

85 

3.04 

446 

27 

3.60 

609 

58 

2.85 

381 




28 

3.37 

559 

59 

2.90 

384 




29 

3.80 

521 







30 

3.76 

646 







31 

3.24 

467 








(b) Calculate W _1 and B and the eigenvalues and eigenvectors of W^B. Use the linear 
discriminants derived from these eigenvectors to classify the new observation 
xq = [3.21 497] into one of the populations 7 r 2 : admit; 7 r 2 : not admit; and 7 t 3 : bor¬ 

derline. Does the classification agree with that in Example 11.11? Should it? Explain. 

11.30. Gerrild and Lantz [13] chemically analyzed crude-oil samples from three zones of sandstone: 
77 1 *. Wilhelm 
7 t 2 : Sub-Mulinia 
7 t 3 : Upper 

The values of the trace elements 

Xi — vanadium (in percent ash) 

X 2 = iron (in percent ash) 

X 3 = beryllium (in percent ash) 
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and two measures of hydrocarbons, 

X 4 = saturated hydrocarbons (in percent area) 

X 5 = aromatic hydrocarbons (in percent area) 

are presented for 56 cases in Table 11.7.The last two measurements are determined from! 

areas under a gas-liquid chromatography curve. 

(a) Obtain the estimated minimum TPM rule, assuming normality. Comment on tE$| 

adequacy of the assumption of normality. ~fg 

(b) Determine the estimate of £(AER) using Lachenbruch’s holdout procedures 
give the confusion matrix. 

(c) Consider various transformations of the data to normality (see Example Il.l4),an3§j 
repeat Parts a and b. 


Table 11-7 Crude-Oil Data 



*1 

*2 

*3 

x 4 

*5 

TTx 

3.9 

51.0 

0.20 


12.19 

2.7 

49.0 

0.07 

7.14 

12.23 


2.8 

36.0 

0.30 


11.30 


3.1 

45.0 

0.08 

7.20 

13.01 


3.5 

46.0 

0.10 

7.81 

12.63 


3.9 

43.0 

0.07 

6.25 

10.42 


2.7 

35.0 

0.00 

5.11 

9.00 


5.0 

47.0 

0.07 


6.10 

3.4 

32.0 

0.20 

5.82 

4.69 


1.2 

12.0 

0.00 

5.54 

3.15 


8.4 

17.0 

0.07 

6.31 

4.55 


4.2 

36.0 

0.50 

9.25 

4.95 


4.2 

35.0 

0.50 

5.69 

2.22 


3.9 

41.0 

0.10 

5.63 

2.94 


3.9 

36.0 

0.07 

6.19 

2.27 


7.3 

32.0 

0.30 


12.92 


4.4 

46.0 

0.07 

7.54 

5.76 


3.0 

30.0 

0.00 

5.12 

10.77 

^3 

6.3 

13.0 

0.50 

4.24 

8.27 

1.7 

5.6 

1.00 

5.69 

4.64 


7.3 

24.0 

0.00 

4.34 

2.99 


7.8 

18.0 

0.50 

3.92 

6.09 


7.8 

25.0 

0.70 

5.39 

6.20 


7.8 

26.0 

1.00 

5.02 

2.50 


9.5 

17.0 

0.05 

3.52 

5.71 


7.7 

14.0 

0.30 

4.65 

8.63 


11.0 

20.0 

0.50 

4.27 

8.40 


8.0 

14.0 

0.30 

4.32 

7.87 


8.4 

18.0 

0.20 

4.38 

7.98 






(continues on next page) 


Table 11.7 (continued) 

*i 

*2 

*3 

*4 

*5 

10.0 

18.0 

0.10 

3.06 

7.67 

7.3 

15.0 

0.05 

3.76 

6.84 

9.5 

22.0 

0.30 

3.98 

5.02 

8.4 

15.0 

0.20 

5.02 

10.12 

8.4 

17.0 

0.20 

4.42 

8.25 

9.5 

25.0 

0.50 

4.44 

5.95 

7.2 

22.0 

1.00 

4.70 

3.49 

4.0 

12.0 

0.50 

5.71 

6.32 

6.7 

52.0 

0.50 

4.80 

3.20 

9.0 

27.0 

0.30 

3.69 

3.30 

7.8 

29.0 

1.50 

6.72 

5.75 

4.5 

41.0 

0.50 

3.33 

2.27 

6.2 

34.0 

0.70 

7.56 

6.93 

5.6 

20.0 

0.50 

5.07 

6.70 

9.0 

17.0 

0.20 

4.39 

8.33 

8.4 

20.0 

0.10 

3.74 

3.77 

9.5 

19.0 

0.50 

3.72 

7.37 

9.0 

20.0 

0.50 

5.97 

11.17 

6.2 

16.0 

0.05 

4.23 

4.18 

7.3 

20.0 

0.50 

4.39 

3.50 

3.6 

15.0 

0.70 

7.00 

4.82 

6.2 

34.0 

0.07 

4.84 

2.37 

7.3 

22.0 

0.00 

4.13 

2.70 

4.1 

29.0 

0.70 

5.78 

7.76 

5.4 

29.0 

0.20 

4.64 

2.65 

5.0 

34.0 

0.70 

4.21 

6.50 

6.2 

27.0 

0.30 

3.97 

2.97 


11.31. Refer to the data on-salmon in Table 11.2. 

(a) Plot the bivariate data for the two groups of salmon. Are the sizes and orientation of 
the scatters roughly the same? Do bivariate normal distributions with a common co- 
variance matrix appear to be viable population models for the Alaskan and Canadi¬ 
an salmon? 

(b) Using a linear discriminant function for two normal populations with equal priors 
and equal costs [see (11-19)], construct dot diagrams of the discriminant scores for 
the two groups. Does it appear as if the growth ring diameters separate for the two 
groups reasonably well? Explain. 

(c) Repeat the analysis in Example 11.8 for the male and female salmon separately. Is it 
easier to discriminate Alaskan male salmon from Canadian male salmon than it is to 
discriminate the females in the two groups? Is gender (male or female) likely to be a 
useful discriminatory variable? 

11.32. Data on hemophilia A carriers, similar to those used in Example 11.3, are listed in 

Table 11.8 on page 664. (See [15].) Using these data, 

(a) Investigate the assumption of bivariate normality for the two groups. 
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(b) Obtain the sample linear discriminant function, assuming equal prior probabilities, 
and estimate the error rate using the holdout procedure. ‘ 

(c) Classify the following 10 new cases using the discriminant function in Part b. 

(d) Repeat Parts a-c, assuming that the prior probability of obligatory carriers (group 2) 
is j and that of noncarriers (group 1) is 


New Cases Requiring Classification 


Case 

log J0 (AHF activity) 

log 10 (AHF antigen) 

1 

-.112 

-.279 

2 

-.059 

-.068 

3 

.064 

.012 

4 

-.043 

-.052 

5 

-.050 

-.098 

6 

-.094 

-.113 

7 

-.123 

-.143 

8 

-.011 

-.037 

9 

-.210 

-.090 

10 

-.126 

-.019 


11.33. Consider the data on bulls in Table 1.10. 

(a) Using the variables YrHgt, FtFrBody, PrctFFB, Frame, BkFat, SaleHt, and SaleWt, 
calculate Fisher s linear discriminants, and classify the bulls as Angus, Hereford, 
or Simental. Calculate an estimate of ^(AER) using the holdout procedure. 
Classify a bull with characteristics YrHgt = 50, FtFrBody = 1000, PrctFFB = 73, 
Frame = 7, BkFat — .17, SaleHt = 54, and SaleWt = 1525 as one of the three 
breeds. Plot the discriminant scores for the bulls in the two-dimensional discriminant 
space using different plotting symbols to identify the three groups. 

(b) Is there a subset of the original seven variables that is almost as good for discrimi¬ 
nating among the three breeds? Explore this possibility by computing the estimated 
£(AER) for various subsets. 

11.34. Table 11.9 on pages 666-667 contains data on breakfast cereals produced by three 
different American manufacturers: General Mills (G), Kellogg (K), and Quaker (Q). 
Assuming multivariate normal data with a common covariance matrix, equal costs, and 
equal priors, classify the cereal brands according to manufacturer. Compute the estimat¬ 
ed £“(AER) using the holdout procedure. Interpret the coefficients of the discriminant 
functions. Does it appear as if some manufacturers are associated with more “nutritional” 
cereals (high protein, low fat, high fiber, low sugar, and so forth) than others? Plot the 
cereals in the two-dimensional discriminant space, using different plotting symbols to 
identify the three manufacturers. 

11.35. Table 11.10 on page 668 contains measurements on the gender, age, tail length (mm), and 
snout to vent length (mm) for Concho Water Snakes. 

Define the variables 

X x = Gender 
X 2 = Age 
X 2 = TailLength 
X 4 = SntoVnLength 





667 


ON 

On 

On 


1 Apple_Cinnamon_Cheerios 

2 Cheerios 

3 Cocoa_Puffs 

4 Count_Chocula 

5 Golden_Grahams 

6 Honey_Nut_Cheerios 

7 Kix 

8 Lucky_Charms 

9 MultLGrain„Cheerios 

10 Oatmeal_Raisin_Crisp 

11 Raisin_Nut_Bran 

12 TotaLCorn^Fiakes 

13 Total_Raisin_Bran 

14 Total_Whoie_Grain 

15 Trix 

16 Wheaties 

17 Wheaties_Honey_Goid 

18 AllJBran 

19 Applejacks 

20 Corn_Flakes 

21 Corn_Pops 


G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

K 

K 

K 

K 


Tahlp 110 Data on Rrands of Cereal ---—-- 

Brand 

Manufacturer 

Calories Protein Fat Sodium Fiber Carbohydrates Sugar Potassium Group 
_ ~ i on ic me in 70 1 


110 

110 

110 

110 

110 

110 

110 

100 

130 

100 

110 

140 

100 

110 

100 

110 

70 

110 

100 

110 


6 

1 

1 

1 

3 

2 

2 

2 

3 

3 

2 

3 

3 

1 

3 
2 

4 
2 
2 
1 


2 

1 

1 

1 

1 

1 

1 

1 

2 

2 

1 

1 

1 

1 

1 

1 

1 

0 

0 

0 


290 

180 

180 

280 

250 

260 

180 

220 

170 

140 

200 

190 

200 

140 

200 

200 

260 

125 

290 

90 


2.0 

0.0 

0.0 

0.0 

1.5 

0.0 

0.0 

2.0 

1.5 

2.5 
0.0 
4.0 
3.0 
0.0 
3.0 
1.0 
9.0 
1.0 
1.0 
1.0 


17.0 

12.0 

12.0 

15.0 

11.5 
21.0 
12.0 
15.0 

13.5 

10.5 
21.0 
15.0 
16.0 
13.0 
17.0 
16.0 

7.0 

11.0 

21.0 

13.0 


1 

13 

13 

9 
10 

3 

12 

6 

10 
8 
3 

14 
3 

12 
3 
8 
. 5 
14 
2 
12 


105 

55 

65 

45 

90 

40 

55 

90 

120 

140 

35 

230 

110 

25 

110 

60 

320 

30 

35 

20 


1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
1 
2 
2 
2 
2 

continued 







22 Cracklin’_OatJBran 

K 

110 

3 

3 

140 

4.0 

10.0 

7 

160 

2 

23 Crispix 

K 

110 

2 

0 

220 

1.0 

21.0 

3 

30 

2 

24 FrootJLoops 

K 

110 

2 

1 

125 

1.0 

11.0 

13 

30 

2 

25 Frosted_Flakes 

K 

110 

1 

0 

200 

1.0 

14.0 

11 

25 

2 

26 Frosted_Mini_Wheats 

K 

100 

3 

0 

0 

3.0 

14.0 

7 

100 

2 

27 Fruitful_Bran 

K 

120 

3 

0 

240 

5.0 

14.0 

12 

190 

2 

28 Just_Right_Crunchy_Nuggets 

K 

110 

2 

1 

170 

1.0 

17.0 

6 

60 

2 

29 Mueslix_Crispy_Blend 

K 

160 

3 

2 

150 

3.0 

17.0 

13 

160 

2 

30 Nut&Honey_Crunch 

K 

120 

2 

1 

190 

0.0 

15.0 

9 

40 

2 

31 Nutri-grain_Almond-Raisin 

K 

140 

3 

2 

220 

3.0 

21.0 

7 

130 

2 

32 Nutri-grain_Wheat 

K 

90 

3 

0 

170 

3.0 

18.0 

2 

90 

2 

33 Product_19 

K 

100 

3 

0 

320 

1.0 

20.0 

3 

45 

2 

34 Raisin Bran 

K 

120 

3 

1 

210 

5.0 

14.0 

12 

240 

2 

35 Rice_Krispies 

K 

110 

2 

0 

290 

0.0 

22.0 

3 

35 

2 

36 Smacks 

K 

110 

2 

1 

70 

1.0 

9.0 

15 

40 

2 

37 Special_K 

K 

110 

6 

0 

230 

1.0 

16.0 

3 

55 

2 

38 Cap’n’Crunch 

Q 

120 

1 

2 

220 

0.0 

12.0 

12 

35 

3 

39 Honey_Graham_Ohs 

Q 

120 

1 

2 

220 

1.0 

12.0 

11 

45 

3 

40 Life 

Q 

100 

4 

' 2 

150 

2.0 

12.0 

6 

95 

3 

41 Puffed_Rice 

' Q 

50 

1 

0 

0 

0.0 

13.0 

0 

15 

3 

42 Puffed_Wheat 

Q 

50 

2 

0 

0 

1.0 

10.0 

0 

50 

3 

43 Quaker_Oatmeal 

Q 

100 

5 

2 

0 

2.7 

1.0 

1 

110 

3 

Source: Data courtesy of Chad Dacus. 
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Table ILIO Concho Water Snake Data 


Gender 

Age 

TailLength 

Snto 

VnLength 


Gender 

Age 

TailLength 

Snto 

VnLength 

1 

Female 

2 

127 

441 

1 

Male 

2 

126 

457 

2 

Female 

2 

171 

455 

2 

Male 

2 

128 

466 

3 

Female 

2 

171 

462 

3 

Male 

2 

151 

466 

4 

Female 

2 

164 

446 

4 

Male 

2 

115 

361 

5 

Female 

2 

165 

463 

5 

Male 

2 

138 

473 

6 

Female 

2 

127 

393 

6 

Male 

2 

145 

477 

7 

Female 

2 

162 

451 

7 

Male 

3 

145 

507 

8 

Female 

2 

133 

376 

8 

Male 

3 

145 

493 

9 

Female 

2 

173 

475 

9 

Male 

3 

158 

558 

10 

Female 

2 

145 

398 

10 

Male 

3 

152 

495 

11 

Female 

2 

154 

435 

11 

Male 

3 

159 

521 

12 

Female 

3 

165 

491 

12 

Male 

3 

138 

487 

13 

Female 

3 

178 

485 

13 

Male 

3 

166 

565 

14 

Female 

3 

169 

477 

14 

Male 

3 

168 

585 

15 

Female 

3 

186 

530 

15 

Male 

3 

160 

550 

16 

Female 

3 

170 

478 

16 

Male 

4 

181 

652 

17 

Female 

3 

182 

511 

17 

Male 

4 

185 

587 

18 

Female 

3 

172 

475 

18 

Male 

4 

172 

606 

19 

Female 

3 

182 

487 

19 

Male 

4 

180 

591 

20 

Female 

3 

172 

454 

20 

Male 

4 

205 

683 

21 

Female 

3 

183 

502 

21 

Male 

4 

175 

625 

22 

Female 

3 

170 

483 

22 

Male 

4 

182 

612 

23 

Female 

3 

171 

477 

23 

Male 

4 

185 

618 

24 

Female 

3 

181 

493 

24 

Male 

4 

181 

613 

25 

Female 

3 

167 

490 

25 

Male 

4 

167 

600 

26 

Female 

3 

175 

493 

26 

Male 

4 

167 

602 

27 

Female 

3 

139 

477 

27 

Male 

4 

160 

596 

28 

Female 

3 

183 

501 

28 

Male 

4 

165 

611 

29 

Female 

4 

198 

537 

29 

Male 

4 

173 

603 

30 

Female 

4 

190 

566 






31 

Female 

4 

192 

569 






32 

Female 

4 

211 

574 






33 

Female 

4 

206 

570 






34 

Female 

4 

206 

573 






35 

Female 

4 

165 

531 






36 

Female 

4 

189 

528 






37 

Female 

4 

195 

536 






Source: Data courtesy of Raymond J. Carroll. 


(a) Plot the data as a scatter plot with tail length (jc 3 ) as the horizontal axis and snout to 
vent length (x 4 ) as the vertical axis. Use different plotting symbols for female and 
male snakes, and different symbols for different ages. Does it appear as if tail length 
and snout to vent length might usefully discriminate the genders of snakes? The dif¬ 
ferent ages of snakes? 

(b) Assuming multivariate normal data with a common covariance matrix, equal priors, 
and equal costs, classify the Concho Water Snakes according to gender. Compute the 
estimated £(AER) using the holdout procedure. 


(c) Repeat part (b) using age as the groups rather than gender. 

(d) Repeat part (b) using only snout to vent length to classify the snakes according to 
age. Compare the results with those in part (c). Can effective classification be 
achieved with only a single variable in this case? Explain. 

11.36. Refer to Example 11.17. Using logistic regression, refit the salmon data in Table 11.2 
with only the covariates freshwater growth and marine growth. Check for the signifi¬ 
cance of the model and the significance of each individual covariate. Set a = .05. Use 
the fitted function to classify each of the observations in Table 11.2 as Alaskan salmon or 
Canadian salmon using rule (11-77). Compute the apparent error rate, APER, and com¬ 
pare this error rate with the error rate from the linear classification function discussed in 
Example 11.8. 
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12J Introduction 

Rudimentary, exploratory procedures are often quite helpful in understanding 
the complex nature of multivariate relationships. For example, throughout 
this book, we have emphasized the value of data plots. In this chapter, we shall dis¬ 
cuss some additional displays based on certain measures of distance and suggested 
step-by-step rules (algorithms) for grouping objects (variables or items). Searching 
the data for a structure of “natural” groupings is an important exploratory 
technique. Groupings can provide an informal means for assessing dimensionality, 
identifying outliers, and suggesting interesting hypotheses concerning relationships. 

Grouping, or clustering, is distinct from the classification methods discussed in 
the previous chapter. Classification pertains to a known number of groups, and the 
operational objective is to assign new observations to one of these groups. Cluster 
analysis is a more primitive technique in that no assumptions are made concerning 
the number of groups or the group structure. Grouping is done on the basis of simi¬ 
larities or distances (dissimilarities). The inputs required are similarity measures or 
data from which similarities can be computed. 

To illustrate the nature of the difficulty in defining a natural grouping, consider 
sorting the 16 face cards in an ordinary deck of playing cards into clusters of similar 
objects. Some groupings are illustrated in Figure 12.1. It is immediately clear that 
meaningful partitions depend on the definition of similar. 

In most practical applications of cluster analysis, the investigator knows enough 
about the problem to distinguish “good” groupings from “bad” groupings. Why not 
enumerate all possible groupings and select the “best” ones for further study? 
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(a) Individual cards 

4 4 ♦ 4 
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K 

Q 

J _||_ 

(c) Black and red suits 


4 4 4 4 



(e) Hearts plus queen of spades 
and other suits (hearts) 


4 4 4 4 



(b) Individual suits 


4 4 4 4 



(d) Major and minor suits (bridge) 


4 4 4 4 



(f) Like face cards 


Figure 12 J Grouping face cards. 


For the playing-card example, there is one way to form a single group of 
16 face cards, there are 32,767 ways to partition the face cards into two groups (of 
varying sizes), there are 7,141,686 ways to sort the face cards into three groups 
(of varying sizes), and so on. 1 Obviously, time constraints make it impossible to 
determine the best groupings of similar objects from a list of all possible struc¬ 
tures. Even fast computers are easily overwhelmed by the typically large number 
of cases, so one must settle for algorithms that search for good, but not necessarily 
the best, groupings. 

To summarize, the basic objective in cluster analysis is to discover natural 
groupings of the items (or variables). In turn, we must first develop a quantitative 
scale on which to measure the association (similarity) between objects. Section 12.2 
is devoted to a discussion of similarity measures. After that section, we describe a 
few of the more common algorithms for sorting objects into groups. 


1 The number of ways of sorting n objects into k nonempty groups is a Stirling number of the second 
kind given by (1/*!) 2 (See [1].) Adding these numbers for k = 1,2,... ,n groups, we 

obtain the total number of possible ways to sort n objects into groups. 
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Even without the precise notion of a natural grouping, we are often able to 
group objects in two- or three-dimensional plots by eye. Stars and Chernoff faces, 
discussed in Section 1.4, have been used for this purpose. (See Examples 1.11 and 
1.12.) Additional procedures for depicting high-dimensional observations in two di¬ 
mensions such that similar objects are, in some sense, close to one another are con¬ 
sidered in Sections 12.5-12.7. 


12.2 Similarity Measures 

Most efforts to produce a rather simple group structure from a complex data set re¬ 
quire a measure of “closeness,” or “similarity.” There is often a great deal of subjec¬ 
tivity involved in the choice of a similarity measure. Important considerations 
include the nature of the variables (discrete, continuous, binary), scales of measure¬ 
ment (nominal, ordinal, interval, ratio), and subject matter knowledge. 

When items (units or cases) are clustered, proximity is usually indicated by 
some sort of distance. By contrast, variables are usually grouped on the basis of 
correlation coefficients or like measures of association. 


Distances and Similarity Coefficients for Pairs of Items 

We discussed the notion of distance in Chapter 1, Section 1.5. Recall that the 
Euclidean (straight-line) distance between two p-dimensional observations (items) 
x' = [xi, x 2$ -.. ■> x p ] and y' = [y u ..., y p ] is, from (1-12), 

d(x,y) = Vo*! - Jy + (x 2 - >- 2 ) 2 + ■■ ■+ (x p - y p ) 2 

= V(x -y)'(x - y) (12-1) 


The statistical distance between the same two observations is of the form [see (1-23)] 


y) = V(x - y) # A(x - y) 


Ordinarily, A = S“\ where S contains the sample variances and covariances. 
However, without prior knowledge of the distinct groups, these sample quantities 
cannot be computed. For this reason, Euclidean distance is often preferred for 
clustering. 

Another distance measure is the Minkowski metric 


P ’ll /m 

d(x,y) = 2k-*1" (12-3) 

_/=l 

For m ~ 1, d(x,y) measures the “city-block” distance between two points in p 
dimensions. For m = 2, d(x,y) becomes the Euclidean distance. In general, varying 
m changes the weight given to larger and smaller differences. 
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Two additional popular measures of “distance” or dissimilarity are given by the 
Canberra metric and the Czekanowski coefficient Both of these measures are 


defined for nonnegative variables only. We have 

p 

1 x i “ 1 


Canberra metric: d(x, y) ~ 2 

i= 1 

(*,■ + yd 

P 

2 £ rnin(x,, yd 

(12-4) 

Czekanowski coefficient: d(x, y) = 1 — 

1=1 

2 ( x i + yd 

i= 1 

(12-5) 


Whenever possible, it is advisable to use “true” distances that is, distances satisfy¬ 
ing the distance properties of (1-25)—for clustering objects. On the other hand, 
most clustering algorithms will accept subjectively assigned distance numbers that 
may not satisfy, for example, the triangle inequality. 

When items cannot be represented by meaningful p-dimensional measure¬ 
ments, pairs of items are often compared on the basis of the presence or absence of 
certain characteristics. Similar items have more characteristics in common than do 
dissimilar items. The presence or absence of a characteristic can be described 
mathematically by introducing a binary variable , which assumes the value 1 if the 
characteristic is present and the value 0 if the characteristic is absent. For p — 5 
binary variables, for instance, the “scores” for two items i and k might be arranged as 
follows: 




Variables 




1 

2 

3 

4 

5 

Item i 

1 

0 

0 

1 

1 

Item k 

1 

1 

0 

1 

0 


In this case, there are two 1-1 matches, one CM) match, and two mismatches. 

Let Xij be the score (1 orO) of the ;th binary variable on the ith item and x kJ be the 
score (again, 1 or 0) of the ;'th variable on the kth item,; = 1,2,..., p. Consequently, 


( x ij ~ x k}) 2 


0 if Xij = x kj = 1 or x- t j = x k} = 0 
1 if Xij * x kj 


( 12 - 6 ) 


p 

and the squared Euclidean distance, ^ {x { j - x k j) 2 , provides a count of the number 

of mismatches. A large distance corresponds to many mismatches—that is, dissimi¬ 
lar items. From the preceding display, the square of the distance between items i and 
k would be 


£ (x,j - x k j) 2 = (1 - l) 2 + (0 - l) 2 + (0 - 0) 2 + (1 - l) 2 + (1 - 0) 2 
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Although a distance based on (12-6) might be used to measure similarity, it suf¬ 
fers from weighting the 1-1 and CM) matches equally. In some cases, a 1-1 match is a 
stronger indication of similarity than a CM) match. For instance, in grouping people, 
the evidence that two persons both read ancient Greek is stronger evidence of simi¬ 
larity than the absence of this ability. Thus, it might be reasonable to discount the 
0-0 matches or even disregard them completely.To allow for differential treatment 
of the 1-1 matches and the 0-0 matches, several schemes for defining similarity co¬ 
efficients have been suggested. 

To introduce these schemes, let us arrange the frequencies of matches and mis¬ 
matches for items i and k in the form of a contingency table: 


(12-7) 


In this table, a represents the frequency of 1-1 matches, b is the frequency of 1-0 
matches, and so forth. Given the foregoing five pairs of binary outcomes, <2 = 2 and 
b = c = d = 1. 

Table 12.1 lists common similarity coefficients defined in terms of the frequen¬ 
cies in (12-7). A short rationale follows each definition. 
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Coefficients 1 2, and 3 in the table are monotonically related Suppose 
coefficient 1 is calculated for two contingency tables. Table I a nd Tab e . Then 

if (a + dMD > lau + d u )/p, we also have 2(fli + d { )/[2(a x + fli) qj 

i V/' + a /P wr 2 L + du) + fcn + c,,]. and coefficient 3 will be at least as large 

for Table I as kis for Table II. (See Exercise 12.4.) Coefficients 5,6, and 7 also re- 

tai " Monotolritily is taaportant, because so m e clustering procedures are not affected 
if ^ definition of similarity is changed in a manner that leaves the relative order,ngs 
of similarities unchanged. The single linkage and complete linkage hierarchical 
procedures Scussed in Section 12.3 are not affected. For these methods, any cho.ce 
of the coefficients l,2,and3 in Table 12.1 will produce the same groupings. Similarly, 
any choice of the coefficients 5,6, and 7 will yield identical groupings. 


Example 12.1 (Calculating the values of a similarity coefficient) Suppose five indi¬ 
viduals possess the following characteristics. 



Height 

Weight 

Eye 

color 

Hair 

color 

Handedness 

Gender 

Individual 1 
Individual 2 
Individual 3 
Individual 4 
Individual 5 

68 in 

73 in 

67 in 

64 in 

76 in 

1401b 
185 lb 
165 lb 
120 lb 
2101b 

green 

brown 

blue 

brown 

brown 

blond 

brown 

blond 

brown 

brown 

right 

right 

right 

right 

left 

female 

male 

male 

female 

male 


Define six binary variables Xj , X 2 , X 3 ,X 4 , X*,, X 6 as 

[l heights 72 in. = U blond hair 

Xl= |0 height <72 in. 4 \o not blond h; 

(l weights 1501b = 0 right hande< 

Xl = \o weight < 1501b 5 \0 lefthanded 

j 1 brown eyes ^ _ |l female 

Xl ~ [0 otherwise 6 male 

The scores for individuals 1 and 2 on the p = 6 binary variables are 


1 blond hair 
0 not blond hair 

1 right handed 
0 left handed 

[ 1 female 
10 male 


Individual 1 0 ? ? n 1 0 

2 1 1 1 0 1 u 

and the number of matches and mismatches are indicated in the two-way array 

Individual 2 
] l 0 I Total 


Individual 1 


1 

0 

Totals 
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Employing similarity coefficient 1, which gives equal weight to matches, we 
compute 

fl + d = 1 + 0 = 1 
p 6 6 

Continuing with similarity coefficient 1, we calculate the remaining similarity 
numbers for pairs of individuals. These are displayed in the 5X5 symmetric 
matrix 

Individual 
1 2 3 4 5 

1 

2 

Individual 

3 

4 

5 

Based on the magnitudes of the similarity coefficient, we should conclude that 
individuals 2 and 5 are most similar and individuals 1 and 5 are least similar. Other 
pairs fall between these extremes. If we were to divide the individuals into two rela¬ 
tively homogeneous subgroups on the basis of the similarity numbers, we might 
form the subgroups (1 3 4) and (2 5). 

Note that X 3 = 0 implies an absence of brown eyes, so that two people, one 
with blue eyes and one with green eyes, will yield a 0-0 match. Consequently, it may 
be inappropriate to use similarity coefficient 1,2, or 3 because these coefficients give 
the same weights to 1-1 and 0-0 matches. ■ 



We have described the construction of distances and similarities. It is always 
possible to construct similarities from distances. For example, we might set 

= ttV < 12 ’ 8) 

1 + d lk 

where 0 < l ik < 1 is the similarity between items i and k and d ik is the corre¬ 
sponding distance. 

However, distances that must satisfy (1-25) cannot always be constructed from 
similarities. As Gower [11,12] has shown, this can be done only if the matrix of sim¬ 
ilarities is nonnegative definite. With the nonnegative definite condition, and with 
the maximum similarity scaled so that T/, = 1, 


d ik = v 2(1 


has the properties of a distance. 


Similarities and Association Measures for Pairs of Variables 

Thus far, we have discussed similarity measures for items. In some applications, it is 
the variables, rather than the items, that must be grouped. Similarity measures for 
variables often take the form of sample correlation coefficients. Moreover, in some 
clustering applications, negative correlations are replaced by their absolute values. 
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When the variables are binary, the data can again be arranged in the form of a 
contingency table. This time, however, the variables, rather than the items, delineate 
the categories. For each pair of variables, there are n items categorized in the table. 
With the usual 0 and 1 coding, the table becomes as follows: 

Variable k 


(12-10) 



For instance, variable i equals 1 and variable k equals 0 for b of the n items. 

The usual product moment correlation formula applied to the binary variables 
in the contingency table of (12-10) gives (see Exercise 12.3) 


ad - be 

[(a + b){c + d)(a + c)(b + d)}' 12 


( 12 - 11 ) 


This number can be taken as a measure of the similarity between the two variables. 

The correlation coefficient in (12-11) is related to the chi-square statistic 
(r 2 =.x z /n) for testing the independence of two categorical variables. For n fixed, a 
large similarity (or correlation) is consistent with the presence of dependence. 

Given the table in (12-10), measures of association (or similarity) exactly analo¬ 
gous to the ones listed in Table 12.1 can be developed. The only change required is 
the substitution of n (the number of items) for p (the number of variables). 


Concluding Comments on Similarity 

To summarize this section, we note that there are many ways to measure the simi¬ 
larity between pairs of objects. It appears that most practitioners use distances [see 
(12-1) through (12-5)] or the coefficients in Table 12.1 to cluster items and correla¬ 
tions to cluster variables . However, at times, inputs to clustering algorithms may be 
simple frequencies. 

Example 12.2 (Measuring the similarities of 11 languages) The meanings of words 
change with the course of history. However, the meaning of the numbers 1, 2, 3,... 
represents one conspicuous exception. Thus, a first comparison of languages might 
be based on the numerals alone. Table 12.2 gives the first 10 numbers in English, 
Polish, Hungarian, and eight other modem European languages. (Only languages 
that use the Roman alphabet are considered, and accent marks, cedillas, diereses, 
etc., are omitted.) A cursory examination of the spelling of the numerals in the table 
suggests that the first five languages (English, Norwegian, Danish, Dutch, and Ger¬ 
man) are very much alike. French, Spanish, and Italian are in even closer agreement. 
Hungarian and Finnish seem to stand by themselves, and Polish has some of the 
characteristics of the languages in each of the larger subgroups. 
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Table 12.3 Concordant First Letters for Numbers in 11 Languages 


E 

N 

Da 

Du 

G 

Fr 

Sp 

I 

P 

H Fi 

E 

10 










N 

8 

10 









Da 

8 

9 

10 








Du 

3 

5 

4 

10 







G 

4 

6 

5 

5 

10 






Fr 

4 

4 

4 

1 

3 

10 





Sp 

4 

4 

5 

1 

3 

8 

10 




I 

4 

4 

5 

1 

3 

9 

9 

10 



P 

3 

3 

4 

0 

2 

5 

7 

6 

10 


H 

1 

2 

2 

2 

1 

0 

0 

0 

0 

10 

Fi 

1 

1 

1 

1 

1 

1 

1 

1 

1 

2 10 


The words for 1 in French, Spanish, and Italian all begin with u. For illustrative 
purposes, we might compare languages by looking at the first letters of the numbers. 
We call the words for the same number in two different languages concordant if they 
have the same first letter and discordant if they do not. From Table 12.2, the table of 
concordances (frequencies of matching first initials) for the numbers 1-10 is given in 
Table 12.3: We see that English and Norwegian have the same first letter for 8 of the 
10 word pairs. The remaining frequencies were calculated in the same manner. 

The results in Table 12.3 confirm our initial visual impression of Table 12.2. That 
is, English, Norwegian, Danish, Dutch, and German seem to form a group. French, 
Spanish, Italian, and Polish might be grouped together, whereas Hungarian and 
Finnish appear to stand alone. ■ 

In our examples so far, we have used our visual impression of similarity or dis¬ 
tance measures to form groups. We now discuss less subjective schemes for creating 
dusters. 


12.3 Hierarchical Clustering Methods 

We can rarely examine all grouping possibilities, even with the largest and fastest 
computers. Because of this problem, a wide variety of clustering algorithms have 
emerged that find “reasonable” clusters without having to look at all configurations. 

Hierarchical clustering techniques proceed by either a series of successive 
mergers or a series of successive divisions. Agglomerative hierarchical methods start 
with the individual objects. Thus, there are initially as many clusters as objects. The 
most similar objects are first grouped, and these initial groups are merged according 
to their similarities. Eventually, as the similarity decreases, all subgroups are fused 
into a single cluster. 

Divisive hierarchical methods work in the opposite direction. An initial single 
group of objects is divided into two subgroups such that the objects in one subgroup 
are “far from” the objects in the other. These subgroups are then further divided 
into dissimilar subgroups; the process continues until there are as many subgroups 
as objects—that is, until each object forms a group. 
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3 Merge clusters U and V. Label the newly formed cluster (UV). Update the en- 
tries ,n the distance matrix by (a) deleting the rows and columns corresponding 
to clusters U and V and (b) adding a row and column giving the distances be¬ 
tween cluster (UV) and the remaining clusters. 

4 Reneat Steps 2 and 3 a total of N - 1 times. (All objects will be in a single 
dusterafter the algorithm terminates.) Record the identity of clusters that 
are merged and the levels (distances or similarities) at which the mergers^take 

place. 

The ideas behind any clustering procedure are probably best conveyed through 
examples, which we shall present after brief discussions of the input and algorithmic 
components of the linkage methods. 

Single Linkage 

The inputs to a single linkage algorithm can be distances or similarities between 
nairs'of objects Groups are formed from the individual entities by merging nearest 
nelghbors where the term nearest neighbor connotes the smallest distance or largest 

Simi Suy we must find the smallest distance in D = {d k } and merge the 
corresponding objects, say, V and V, to get the cluster (UV).For Step 3 of the general 
algorithm of (12-12), the distances between (UV) and any other cluster are 

computed by 

dyjv) w = min {dyw^vw} (12-13) 

Here the quantities d uw and d vw are the distances between the nearest neighbors 
of clusters U and Wand clusters V and W, respectively. . 

Se results of single linkage clustering can be graphically displayed in the form 
of a dendrogram, or tree diagram. The branches in the tree represent clusters. The 
branches come together (merge) at nodes whose positions along a distance (or 
similarity indicate the level at which the fusions occur. Dendrograms for some 
specific cases are considered in the following examples. 


Fvamnle 12 3 (Clustering using single linkage) To illustrate the single linkage 
algorithm, we consider the hypothetical distances between pairs of five objects as 

follows: 


1 

D = {4*} = 3 

4 

5 


1 

0 

9 

3 

6 

11 


2 3 4 5 

0 

7 0 

5 9 0 

10 ( 2 ) 8 0 , 


Treating each object as a cluster, we commence clustering by merging the two 
closest items. Since 


min (d ik ) = d 53 - 2 
i.k 
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objects 5 and 3 are merged to form the cluster (35). To implement the next level of 
clustering, we need the distances between the cluster (35) and the remaining objects, 
1,2, and 4. The nearest neighbor distances are 

*7(35)i = min {<7 3l , d sx } = min {3,11} = 3 
d{ 35)2 = min (<7 32 , ^ 52 } = min (7, 10} = 7 
<7(35)4 = min{<7 3 4,<7 54 } = min{9, 8} = 8 

Deleting the rows and columns of D corresponding to objects 3 and 5, and adding a 
row and column for the cluster (35), we obtain the new distance matrix 


(35) 

1 

2 

4 


(35) 1 2 



7 9 0 

8 6 5 



The smallest distance between pairs of clusters is now d^s)i - 3, and we merge 
cluster (1) with cluster (35) to get the next cluster, (135). Calculating 

^( 135)2 = min {rf( 35)2 ^ 12 } = min (7, 9} = 7 
d(i 35)4 “ min {d( 35 ) 4 , 4 } = min {8, 6} = 6 

we find that the distance matrix for the next level of clustering is 

(135) 2 4 
(135) r 0 

2 7 0 

4 L 6 ® 0 

The minimum nearest neighbor distance between pairs of clusters is d 42 = 5, and we 
merge objects 4 and 2 to get the cluster (24). 

At this point we have two distinct clusters, (135) and (24). Their nearest neigh¬ 
bor distance is 


^( 135 )( 24 ) - min {^(i 35 ) 2 j ^( 135 ) 4 } “ min {7,6} - 6 
The final distance matrix becomes 


(135) (24) 

(135) I" 0 
(24) L (D 0 

Consequently, clusters (135) and (24) are merged to form a single cluster of all five 
objects, (12345), when the nearest neighbor distance reaches 6. 

The dendrogram picturing the hierarchical clustering just concluded is shown in 
Figure 12.3. The groupings and the distance levels at which they occur are clearly 
illustrated by the dendrogram. ■ 


In typical applications of hierarchical clustering, the intermediate results— 
where the objects are sorted into a moderate number of clusters—are of chief 
interest. 



J J 
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Figure 12.3 Single linkage 
dendrogram for distances between 
five objects. 


Example 12.4 (Single linkage clustering of II languages) Consider the array of con¬ 
cordances in Table 12.3 representing the closeness between the numbers 1-10 in 11 
languages.To develop a matrix of distances, we subtract the concordances from the 
perfect agreement figure of 10 that each language has with itself. The subsequent 
assignments of distances are 

E N Da Du G Fr Sp I P H Fi 

E To 
N 2 0 

Da 2 © 0 

Du 7 5 6 0 

G 6 4 5 5 0 

Fr 6 6 6 9 7 0 

Sp 665 9720 

I 665 9 7 © © 0 

P 776 10 85340 

H 9 8 8 8 9 10 10 10 10 0 

Fi [9 9 9 9999998 0. 

We first search for the minimum distance between pairs of languages (clusters). 
The minimum distance, 1, occurs between Danish and Norwegian, Italian and 
French, and Italian and Spanish. Numbering the languages in the order in which 
they appear across the top of the array, we have 

^32 ~ ^86 = 1’ and ^87 1 

Since d 16 = 2, we can merge only clusters 8 and 6 or clusters 8 and 7. We cannot 
merge clusters 6, 7, and 8 at level 1. We choose first to merge 6 and 8, and then to 
update the distance matrix and merge 2 and 3 to obtain the dusters (68) and (23). 
Subsequent computer calculations produce the dendrogram in Figure 12.4. 

From the dendrogram, we see that Norwegian and Danish, and also French and 
Italian, cluster at the minimum distance (maximum similarity) level. When the 
allowable distance is increased, English is added to the Norwegian-Danish group, 
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and Spanish merges with the French-Italian group. Notice that Hungarian and 
Finnish are more similar to each other than to the other clusters of languages. How¬ 
ever, these two clusters (languages) do not merge until the distance between nearest 
neighbors has increased substantially. Finally, all the dusters of languages are 
merged into a single cluster at the largest nearest neighbor distance, 9. ■ 


Since single linkage joins clusters by the shortest link between them, the tech¬ 
nique cannot discern poorly separated clusters. [See Figure 12.5(a).] On the other 
hand, single linkage is one of the few clustering methods that can delineate nonel- 
lipsoidal clusters. The tendency of single linkage to pick out long stringlike clusters 
is known as chaining. [See Figure 12.5(b).] Chaining can be misleading if items at 
opposite ends of the chain are, in fact, quite dissimilar. 

Variable 2 Variable 2 



(a) Single linkage confused by near overlap (b) Chaining effect 

Figure 12.5 Single linkage clusters. 

The clusters formed by the single linkage method will be unchanged by any as¬ 
signment of distance (similarity) that gives the same relative orderings as the initial 
distances (similarities). In particular, any one of a set of similarity coefficients from 
Table 12.1 that are monotonic to one another will produce the same clustering. 

Complete Linkage 

Complete linkage clustering proceeds in much the same manner as single linkage 
clusterings, with one important exception: At each stage, the distance (similarity) 
between clusters is determined by the distance (similarity) between the two 
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elements, one from each cluster, that are most distant. Thus, complete linkage 
ensures that all items in a cluster are within some maximum distance (or minimum 
similarity) of each other. 

The general agglomerative algorithm again starts by finding the minimum entry 
in D = {d ik } and merging the corresponding objects, such as U and V , to get cluster 
(UV). For Step 3 of the general algorithm in (12-12), the distances between (UV) 
and any other cluster W are computed by 

d(uv)w = max{d UWy d vw } (12-14) 

Here d uw and d vw are the distances between the most distant members of clusters 
U and W and clusters V and ^respectively. 


n 


Objects 


Figure 12.6 Complete linkage 
dendrogram for distances between 
five objects. 


Example 12.5 (Clustering using complete linkage) Let us return to the distance 
matrix introduced in Example 12.3: 

I 2 3 4 5 

0 

9 0 

3 7 0 

6 5 9 0 

II 10 ( 2 ) 8 0 . 


1 

D = {d ik } = 2 

4 

5 


At the first stage, objects 3 and 5 are merged, since they are most similar. This gives 
the cluster (35). At stage 2, we compute 

rf( 35 )x = max{d 3l ,d 5l } = max {3,11} = 11 
d{ 35)2 = max{^ 32 , d 52 } = 10 
d( 35)4 = max {^ 34 , d 54 } = 9 
and the modified distance matrix becomes 


(35) 

1 

2 

4 


(35) 1 2 4 

0 

11 0 

10 9 0 

9 6 © 0 _ 


The next merger occurs between the most similar groups, 2 and 4, to give the cluster 
(24). At stage 3, we have 


d( 24 ) ( 35 ) - max {^ 2 ( 35 )> ^ 4 { 35 )} _ max{ 10 ,9} 10 


^( 24 )i “ max{d 2 i,d 4l ] — 9 


and the distance matrix 


(35) 

(24) 

0 


10 

0 

11 




The next merger produces the cluster (124). At the final stage, the groups (35) and 
(124) are merged as the single cluster (12345) at level 

«*(124)(35) = max {^i ( 35 ), d(24) (35)} = max{ll,10} = 11 

The dendrogram is given in Figure 12.6. n 

Comparing Figures 12.3 and 12.6, we see that the dendrograms for single link¬ 
age and complete linkage differ in the allocation of object 1 to previous groups. 


Example 12.6 (Complete linkage clustering of I I languages) In Example 12.4, we 
presented a distance matrix for numbers in 11 languages. The complete linkage clus¬ 
tering algorithm applied to this distance matrix produces the dendrogram shown in 
Figure 12.7. 

Comparing Figures 12.7 and 12.4, we see that both hierarchical methods yield the 
English-Norwegian-Danish and the French-Italian-Spanish language groups. Polish is 
merged with French-Italian-Spanish at an intermediate level. In addition, both meth¬ 
ods merge Hungarian and Finnish only at the penultimate stage. 

Howeyer, the two methods handle German and Dutch differently. Single link¬ 
age merges German and Dutch at an intermediate distance, and these two lan¬ 
guages remain a cluster until the final merger. Complete linkage merges German 
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Figure 12.7 Complete linkage 
dendrogram for distances between 
numbers in 11 languages. 
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with the English-Norwegian-Danish group at an intermediate level. Dutch remains 
a cluster by itself until it is merged with the English-Norwegian-Danish-German 
and French -I tali an-Spanish-Polish groups at a higher distance level. The final com¬ 
plete linkage merger involves two clusters. The final merger in single linkage in¬ 
volves three clusters. * 

Example 12.7 (Clustering variables using complete linkage) Data collected on 22 
U.s. public utility companies for the year 1975 are listed in Thble 12.4. Although it is 
more interesting to group companies, we shall see here how the complete linkage al¬ 
gorithm can be used to cluster variables. We measure the similarity between pairs of 


Table 12.4 Public Utility Data (1975) 


Variables 


Company __ x \ A 2 

1. Arizona Public Service 1.06 9.2 

2. Boston Edison Co. 

3. Central Louisiana Electric Co. 1.43 15.4 

4 . Commonwealth Edison Co. ' L02 11.2 

5. Consolidated Edison Co. (N.Y.) 1.49 8.8 

6. Florida Power & Light Co. 1.32 13.5 

7. Hawaiian Electric Co. 1-22 12.2 

8. Idaho Power Co. L1 0 9.2 

9. Kentucky Utilities Co. 1.34 13.0 

10. Madison Gas & Electric Co. 1.12 12.4 

11. Nevada Power Co. -75 7.5 

12. New England Electric Co. 1-13 10.9 

13. Northern States Power Co. 1.15 12.7 

14. Oklahoma Gas & Electric Co. 1.09 12.0 

15. Pacific Gas & Electric Co. -96 7.6 

16. Puget Sound Power & Light Co. 1.16 9.9 

17. San Diego Gas & Electric Co. .76 6.4 

18. The Southern Co. 1-05 12.6 

19. Texas Utilities Co. 116 11.7 

20. Wisconsin Electric Power Co. 1.20 11.8 

21 . United Illuminating Co. L04 8.6 

22. Virginia Electric & Powe r Co. 1-07 9.3 

Key: Xf Fixed-charge coverage ratio (income/debt). 
X 2 : Rate of return on capital. 

X 3 : Cost per KW capacity in place. 

X t -. Annual load factor. 

X s : Peak kWh demand growth from 1974 to 1975. 
X 6 : Sales (kWh use per year). 

Xf. Percent nuclear. 

X s : Total fuel costs (cents per kWh). 

Source: Data courtesy of H. E. Thompson. 


1.6 9077 

2.2 5088 
3.4 9212 


1.0 3300 

-2.2 11127 

2.2 7642 

33 13082 

7.2 8406 

2.7 6455 

6.5 17441 

3.7 6154 

6.4 7179 

1.4 9673 

-0.1 6468 

9.2 15991 

9.0 5714 

2.7 10140 

-2.1 13507 

33 7287 

33 6650 

5.9 10093 


Similarity (correlation) 
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Table 12.5 Correlations Between Pairs of Variables (Public Utility Data) 

*1 "^2 *3 *4 ~X 5 ~X 6 ~X 7 Xg 

" 1.000 

.643 1.000 

-.103 -.348 1.000 

-.082 -.086 .100 1.000 

-.259 -.260 .435 .034 1.000 

-.152 -.010 .028 -.288 .176 1.000 

.045 .211 .115 -.164 -.019 -.374 1.000 

-.013 -.328 .005 .486 -.007 -.561 -.185 1.000 


variables by the product-moment correlation coefficient. The correlation matrix is 
given in Table 12.5. 

When the sample correlations are used as similarity measures, variables with 
large negative correlations are regarded as very dissimilar; variables with large pos¬ 
itive correlations are regarded as very similar. In this case, the “distance” between 
clusters is measured as the smallest similarity between members of the correspond¬ 
ing clusters.The complete linkage algorithm, applied to the foregoing similarity ma¬ 
trix, yields the dendrogram in Figure 12.8. 

We see that variables 1 and 2 (fixed-charge coverage ratio and rate of return on 
capital), variables 4 and 8 (annual load factor and total fuel costs), and variables 3 
and 5 (cost per kilowatt capacity in place and peak kilowatthour demand growth) 
cluster at intermediate “similarity” levels. Variables 7 (percent nuclear) and 6 (sales) 
remain by themselves until the final stages. The final merger brings together the 
(12478) group and the (356) group. ■ 

As in single linkage, a “new” assignment of distances (similarities) that have the 
same relative orderings as the initial distances will not change the configuration of 
the complete linkage dusters. 
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Figure 12.8 Complete linkage 
dendrogram for similarities among 
eight utility company variables. 
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Average Linkage 

Average linkage treats the distance between two clusters as the average distance 
between all pairs of items where one member of a pair belongs to each cluster. 
Again, the input to the average linkage algorithm may be distances or similari¬ 
ties, and the method can be used to group objects or variables.The average linkage 
algorithm proceeds in the manner of the general algorithm of (12-12). We begin by 
searching the distance matrix D = {4*} to find the nearest (most similar) objects— 
for example, U and V .These objects are merged to form the cluster (UV). For Step 
3 of the general agglomerative algorithm, the distances between ( UV) and the other 
cluster W are determined by 

2 2 d ik 

d(uv)w ~ ~ w (12-15) 

N (VV )N W 

where d ik is the distance between object i in the cluster (UV) and object k in the 
cluster W y and N^v) and N w are the number of items in clusters (UV) and W, 
respectively. 


Example 12,8 (Average linkage clustering of 11 languages) The average linkage al¬ 
gorithm was applied to the “distances” between 11 languages given in Example 12.4. 
The resulting dendrogram is displayed in Figure 12.9. 
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6 

4 

2 

0 


A comparison of the dendrogram in Figure 12.9 with the corresponding single 
linkage dendrogram (Figure 12.4) and complete linkage dendrogram (Figure 12.7) 
indicates that average linkage yields a configuration very much like the complete 
linkage configuration. However, because distance is defined differently for each 
case, it is not surprising that mergers take place at different levels. ■ 



Example 12.9 (Average linkage clustering of public utilities) An average linkage 
algorithm applied to the Euclidean distances between 22 public utilities (see 
Table 12.6) produced the dendrogram in Figure 12.10 on page 692. 
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Public utility companies 

Figure 12.10 Average linkage dendrogram for distances between 22 public utility 
companies. 


Concentrating on the intermediate clusters, we see that the utility companies 
tend to group according to geographical location. For example, one intermediate 
cluster contains the firms 1 (Arizona Public Service), 18 (The Southern Company— 
primarily Georgia and Alabama), 19 (Texas Utilities Company), and 14 (Oklahoma 
Gas and Electric Company).There are some exceptions.The cluster (7,12,21,15,2) 
contains firms on the eastern seaboard and in the far west. On the other hand, all 
these firms are located near the coasts. Notice that Consolidated Edison Company 
of New York and San Diego Gas and Electric Company stand by themselves until 
the final amalgamation stages. 

It is, perhaps, not surprising that utility firms with similar locations (or types of 
locations) cluster. One would expect regulated firms in the same area to use, basi¬ 
cally, the same type of fuel(s) for power plants and face common markets. Conse¬ 
quently, types of generation, costs, growth rates, and so forth should be relatively 
homogeneous among these firms. This is apparently reflected in the hierarchical 
clustering. u 

For average linkage clustering, changes in the assignment of distances (similari¬ 
ties) can affect the arrangement of the final configuration of clusters, even though 
the changes preserve relative orderings. 

Ward’s Hierarchical Clustering Method 

Ward [32] considered hierarchical clustering procedures based on minimizing the 
‘loss of information’ from joining two groups. This method is usually implemented 
with loss of information taken to be an increase in an error sum of squares criterion, 



Hierarchical Clustering Methods 693 

ESS. First, for a given cluster k , let ESS*, be the sum of the squared deviations of 
every item in the cluster from the cluster mean (centroid). If there are currently K 
clusters,define ESS as the sum of the ESS* or ESS = ESS t + ESS 2 + ... + ESS*. 
At each step in the analysis, the union of every possible pair of clusters is considered, 
and the two clusters whose combination results in the smallest increase in ESS (min¬ 
imum loss of information) are joined. Initially, each cluster consists of a single item, 
and, if there are N items, ESS* = 0, k = 1, 2,..., N, so ESS = 0. At the other ex¬ 
treme, when all the clusters are combined in a single group of N items, the value of 
ESS is given by 

ESS = 2 (X; - *)'(*,' - x) 

/=1 

where x 7 is the multivariate measurement associated with the yth item and x is the 
mean of all the items. 

The results of Ward’s method can be displayed as a dendrogram. The vertical 
axis gives the values of ESS at which the mergers occur. 

Ward’s method is based on the notion that the clusters of multivariate observa¬ 
tions are expected to be roughly elliptically shaped. It is a hierarchical precursor to 
nonhierarchical clustering methods that optimize some criterion for dividing data 
into a given number of elliptical groups. We discuss nonhierarchical clustering pro¬ 
cedures in the next section. Additional discussion of optimization methods of cluster 
analysis is contained in [8], 


Example 12.10 (Clustering pure malt scotch whiskies) Virtually all the world’s pure 
malt Scotch whiskies are produced in Scotland. In one study (see [22]), 68 binary 
variables were created measuring characteristics of Scotch whiskey that can be 
broadly classified as color, nose, body, palate, and finish. For example, there were 
14 color characteristics (descriptions), including white wine, yellow, very pale, pale, 
bronze, full amber, red, and so forth. LaPointe and Legendre clustered 109 pure malt 
Scotch whiskies, each from a different distillery. The investigators were interested in 
determining the major types of single-malt whiskies, their chief characteristics, and 
the best representative. In addition, they wanted to know whether the groups pro¬ 
duced by the hierarchical clustering procedure corresponded to different geograph¬ 
ical regions, since it is known that whiskies are affected by local soil, temperature, 
and water conditions. 

Weighted similarity coefficients {s,*} were created from binary variables repre¬ 
senting the presence or absence of characteristics. The resulting “distances,” defined 
as {d, k = 1 - 5;*}, were used with Ward’s method to group the 109 pure (single-) 
malt Scotch whiskies. The resulting dendrogram is shown in Figure 12.11. (An aver¬ 
age linkage procedure applied to a similarity matrix produced almost exactly the 
same classification.) 

The groups labelled A-L in the figure are the 12 groups of similar Scotches 
identified by the investigators. A follow-up analysis suggested that these 12 
groups have a large geographic component in the sense that Scotches with similar 
characteristics tend to be produced by distilleries that are located reasonably 
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Figure 12.11 A dendrogram for similarities between 109 pure malt Scotch 
whiskies. 


close to one another. Consequently, the investigators concluded, “The relationship 
with geographic features was demonstrated, supporting the hypothesis that 
whiskies are affected not only by distillery secrets and traditions but also by fac¬ 
tors dependent on region such as water, soil, microclimate, temperature and even 
air quality ” 
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Final Comments—Hierarchical Procedures 

There are many agglomerative hierarchical clustering procedures besides single 
linkage, complete linkage, and average linkage. However, all the agglomerative pro¬ 
cedures follow the basic algorithm of (12-12). 

As with most clustering methods, sources of error and variation are not formal¬ 
ly considered in hierarchical procedures.This means that a clustering method will be 
sensitive to outliers, or “noise points.” 

In hierarchical clustering, there is no provision for a reallocation of objects that 
may have been “incorrectly” grouped at an early stage. Consequently, the final 
configuration of clusters should always be carefully examined to see whether it is 
sensible. 

For a particular problem, it is a good idea to try several clustering methods and, 
within a given method, a couple different ways of assigning distances (similarities). 
If the outcomes from the several methods are (roughly) consistent with one anoth¬ 
er, perhaps a case for “natural” groupings can be advanced. 

The stability of a hierarchical solution can sometimes be checked by applying 
the clustering algorithm before and after small errors (perturbations) have been 
added to the data units. If the groups are fairly well distinguished, the clusterings 
before perturbation and after perturbation should agree. 

Common values (ties) in the similarity or distance matrix can produce multi¬ 
ple solutions to a hierarchical clustering problem. That is, the dendrograms corre¬ 
sponding to different treatments of the tied similarities (distances) can be 
different, particularly at the lower levels. This is not an inherent problem of any 
method; rather, multiple solutions occur for certain kinds of data. Multiple solu¬ 
tions are not necessarily bad, but the user needs to know of their existence so that 
the groupings (dendrograms) can be properly interpreted and different groupings 
(dendrograms) compared to assess their overlap. A further discussion of this issue 
appears in [27]. 

Some data sets and hierarchical clustering methods can produce inversions . 
(See [27].) An inversion occurs when an object joins an existing cluster at a smaller 
distance (greater similarity) than that of a previous consolidation. An inversion is 
represented two different ways in the following diagram: 




(i) (ii) 
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In this example, the clustering method joins A and B at distance 20. At the next 
step, C is added to the group (AB) at distance 32. Because of the nature of the clus¬ 
tering algorithm, D is added to group (ABC) at distance 30, a smaller distance than 
the distance at which C joined (AB). In (i) the inversion is indicated by a dendro¬ 
gram with crossover. In (ii), the inversion is indicated by a dendrogram with a non¬ 
monotonic scale. 

Inversions can occur when there is no clear cluster structure and are generally 
associated with two hierarchical clustering algorithms known as the centroid 
method and the median method. The hierarchical procedures discussed in this book 
are not prone to inversions. 


12.4 Nonhierarchical Clustering Methods 

Nonhierarchical clustering techniques are designed to group items, rather than vari¬ 
ables, into a collection of K clusters. The number of clusters, K may either be speci¬ 
fied in advance or determined as part of the clustering procedure. Because a matrix 
of distances (similarities) does not have to be determined, and the basic data do not 
have to be stored during the computer run, nonhierarchical methods can be applied 
to much larger data sets than can hierarchical techniques. 

Nonhierarchical methods start from either (1) an initial partition of items into 
groups or (2) an initial set of seed points, which will form the nuclei of clusters. 
Good choices for starting configurations should be free of overt biases. One way to 
start is to randomly select seed points from among the items or to randomly parti¬ 
tion the items into initial groups 

In this section, we discuss one of the more popular nonhierarchical procedures, 
the AT-means method. 


/(-means Method 

MacQueen [25] suggests the term K-means for describing an algorithm of his that 
assigns each item to the cluster having the nearest centroid (mean). In its simplest 
version, the process is composed of these three steps: 

1. Partition the items into K initial clusters. 

2. Proceed through the list of items, assigning an item to the cluster whose centroid 
(meaq) is nearest. (Distance is usually computed using Euclidean distance with 
either standardized or unstandardized observations.) Recalculate the centroid 
for the cluster receiving the new item and for the cluster losing the item. 

3. Repeat Step 2 until no more reassignments take place. (12-16) 

Rather than starting with a partition of all items into K preliminary groups 
in Step 1, we could specify K initial centroids (seed points) and then proceed to 
Step 2. 

The final assignment of items to clusters will be, to some extent, dependent 
upon the initial partition or the initial selection of seed points. Experience suggests 
that most major changes in assignment occur with the first reallocation step. 
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Example 12.11 (Clustering using the K-means method) Suppose we measure two 
followingSje 2 e3Ch ° f f ° Ur itemSA B ’ C ’ 3nd D The data are S iven in the 



Observations 

Item 

*1 JC 2 

A 

5 

3 

B 

-1 

1 

C 

1 

-2 

D 

-3 

-2 


The objective is to divide these items into K = 2 clusters such that the 
items within a cluster are closer to one another than they are to the items in 
ifferent clusters. To implement the K = 2-means method, we arbitrarily parti¬ 
tion the items into two clusters, such as (AB) and (CD), and compute the co¬ 
ordinates (■*!, x 2 ) of the cluster centroid (mean). Thus, at Step 1, we have 

I Coordinates of centroid 


Cluster 

*1 

X 2 

(AB) 

5 + (~l) _ 

2 

3 + 1 _ 2 

2 Z 

(CD) 

1 + (-3) 

2 1 

-2 + (-2) 
2 


At Step 2, we compute the Euclidean distance of each item from the group 
centroids and reassign each item to the nearest group. If an item is moved from the 
tmtial configuration, the cluster centroids (means) must be updated before proceed- 
formulas- th C °° rdlnate ’‘ ~ 1-2,..., p, of the centroid is easily updated using the 


nxj + x,j 
n + 1 

nXi Xji 

n - 1 


if the jth item is added to a group 


if the jth item is removed from a group 


Here n is the number of items in the “old” group with centroid x' = (*, x 7 x ) 
Consider the initial clusters (AB) and (CD).The coordinates of the centroids are 
u 1? 2 ’ res P ectlve, y- Suppose item A. with coordinates (5,3) is moved to 

the (CD) group. The new groups are (B) and (ACD) with updated centroids: 


Group (B) 


Group (ACD) x lnew = 


2(2) — 5 

2-1 

2(~l)+5 


Xl, new 


2 ( 2)-3 


= 1 *%n 


2-1 
2(—2) +3 


— 1, the coordinates of B 


= -.33 
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Returning to the initial groupings in Step 1 , we compute the squared distances 

d 2 (A,(AB)) = (5 — 2) 2 + (3 — 2) 2 = 10 if A is not moved 

d 2 (A,(CD)) = (5 + l) 2 + (3 + 2) 2 = 61 

d 2 (A,(B)) = (5 + l) 2 + (3 ~ 1) — 40 if A is moved to the (CD) grou j 

d 2 (A,(ACD)) = (5 - l) 2 + (3 + -33) 2 = 27.09 

Since A is closer to the center of (AB) than it is to the center of (ACD), it is not 
reassigned. 

Continuing, we consider reassigning B. We get 

d 2 (B,(AB)) = (-1 ~ 2) 2 + (1 “ 2) 2 = 10 if B is not moved 
d 2 (B,(CD)) = (-1 + l) 2 + (1 + 2) 2 = 9 

d 2 (B,(A ))) = ( —1 —5) 2 + (1 - 3) 2 = 40 jf Bis moved to the (CD) group 

d 2 (B,(BCD)) = (-1 + l) 2 + (1 + l) 2 = 4 

Since B is closer to the center of (BCD) than it is to the centerof(A S)fiisreas- 
signed to the (CD) group. We now have the clusters (A) and (BCD) with centioid 
coordinates (5,3) and ( — 1, -1) respectively. 

We check C for reassignment. 

d 2 (C,(A)) = (1 — 5) 2 + (—2 — 3) 2 = 41 jfCisnotmoved 

d 2 (C,(BCD)) = (1 + l) 2 + (-2 + l) 2 = 5 

d 2 (C,(AC)) = (1 - 3) 2 + (-2 - -5) 2 = 10.25 jfcis moved to the (A) group 
d 2 (C,(BD)) = (1 + 2) 2 + (-2 + 5) 2 = 11.25 

Since C is closer to the center of the BCD group than it is to the center of the /1C 
group, C is not moved. Continuing in this way, we find that no more reassign 
take place and the final K = 2 clusters are (A) and (BCD). 

For the final clusters, we have 



Squared distances to 
group centroids 


Item 

Cluster 

A B C D 

A 

0 40 41 89 

(BCD) 

52 4 5 .5 


The within cluster sum of squares (sum of squared distances to centroid) are 

Cluster A: ^ 

Cluster (BCD): 4 + 5 + 5 = 14 

Equivalently, we can determine the K = 2 clusters by using the criterion 

min E = 
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where the minimum is over the number of K = 2 clusters and d 2 c (i) is the squared 
distance of case i from the centroid (mean) of the assigned cluster. 

In this example, there are seven possibilities for K — 2 clusters: 

A, (BCD) 

B, (ACD) 

C, (ABD) 

D, (ABC) 

(AB) , (CD) 

(AC) , (BD) 

(AD) , (BC) 

For the A, (BCD) pair: 

A d\ c ( A ) = 0 

(BCD) ^5,c(5) + ^c.c(c) + ^d,c(Z>) = 4 + 5 + 5= 14 

Consequently, 2^(0 = 0 + 14 = 14 
For the remaining pairs, you may verify that 


B, (ACD) 


= 48.7 

C, (ABD) 


= 27.7 

D, (ABC) 


= 31.3 

(AB), (CD) 


= 28 

(AC), (BD) 

o 

= 27 

(AD), (BC) 


= 51.3 


Since the smallest occurs for the pair of clusters (A) and (BCD), this is the 

final partition. ■ 

To check the stability of the clustering, it is desirable to rerun the algorithm with 
a new initial partition. Once clusters are determined, intuitions concerning their in¬ 
terpretations are aided by rearranging the list of items so that those in the first clus¬ 
ter appear first, those in the second cluster appear next, and so forth. A table of the 
cluster centroids (means) and within-cluster variances also helps to delineate group 
differences. 

Example 12.12 {K -means clustering of public utilities) Let us return to the problem 
of clustering public utilities using the data in Table 12.4. The AT-means algorithm for 
several choices of K was run. We present a summary of the results for K — 4 and 
K = 5. In general, the choice of a particular K is not clear cut and depends upon 
subject-matter knowledge, as well as data-based appraisals. (Data-based appraisals 
might include choosing K so as to maximize the between-cluster variability relative 
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to the within-cluster variability. Relevant measures might include |W| 
[see (6-38)] and tr(W _1 B).) The summary is as follows: 


IB + W| 


Number of 
Cluster firms 


Idaho Power Co. (8),Nevada Power Co. (11), Puget 
Sound Power & Light Co. (16), Virginia Electric & 

Power Co. (22), Kentucky Utilities Co. (9). 

Central Louisiana Electric Co. (3), Oklahoma Gas & Electric 
Co. (14),The Southern Co. (18),Texas Utilities Co. (19), 
Arizona Public Service (1), Florida Power & Light Co. (6). 
New England Electric Co. (12), Pacific Gas & Electric 
Co. (15), San Diego Gas & Electric Co. (17), 

United Illuminating Co. (21), Hawaiian Electric Co. (7). 
Consolidated Edison Co. (N.Y.) (5), Boston Edison Co. 

(2), Madison Gas & Electric Co. (10), Northern States 
Power Co. (13), Wisconsin Electric Power Co. 

(20), Commonwealth Edison Co. (4). 


Distances between Cluster Centers 


3.08 0 

3.29 3.56 0 

3.05 2.84 3.18 0_ 


Cluster 


Number of 
firms 


f Nevada Power Co. (11), Puget Sound Power & Light 
< Co. (16), Idaho Power Co. (8), Virginia Electric & Power Co. 

I (22), Kentucky Utilities Co. (9). 

f Central Louisiana Electric Co. (3),Texas Utilities Co. (19), 

<! Oklahoma Gas & Electric Co. (14),The Southern Co. 

I (18), Arizona Public Service (1), Florida Power & Light Co. (6). 

f New England Electric Co. (12), Pacific Gas & Electric 
<J Co. (15), San Diego Gas & Electric Co. (17), United 
l Illuminating Co. (21), Hawaiian Electric Co. (7). 

f Consolidated Edison Co. (N.Y.) (5), Boston 
\ Edison Co. (2). 

/ Commonwealth Edison Co. (4), Madison Gas & Electric Co. (10), 
( Northern States Power Co. (13), Wisconsin Electric Power Co. (20) 
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Distances between Cluster Centers 
1 2 3 4 5 

1 r 0 

2 3.08 0 

3 3.29 3.56 0 

4 3.63 3.46 2.63 0 

5 13.1$ 2.99 3.81 2.89 0 _ 

The cluster profiles (K = 5) shown in Figure 12.12 order the eight variables 
according to the ratios of their between-cluster variability to their within-cluster 
variability. [For univariate F-ratios, see Section 6.4.] We have 

mean square percent nuclear between clusters 3.335 

P = ——----————— ■■——-- = . — 13.1 

nuc mean square percent nuclear within clusters .255 

so firms within different clusters are widely separated with respect to percent nu¬ 
clear, but firms within the same cluster show little percent nuclear variation. Fuel 
costs (FUELC) and annual sales (SALES) also seem to be of some importance in 
distinguishing the clusters. 

Reviewing the firms in the five clusters, it is apparent that the F-means method 
gives results generally consistent with the average linkage hierarchical method. (See 
Example 12.9.) Firms with common or compatible geographical locations cluster. 
Also, the firms in a given cluster seem to be roughly the same in terms of percent 
nuclear. ■ 


We must caution, as we have throughout the book, that the importance of 
individual variables in clustering must be judged from a multivariate perspective. 
All of the variables (multivariate observations) determine the cluster means and 
the reassignment of items. In addition, the values of the descriptive statistics 
measuring the importance of individual variables are functions of the number of 
clusters and the final configuration of the clusters. On the other hand, descriptive 
measures can be helpful, after the fact, in assessing the “success” of the clustering 
procedure. 


Final Comments—Nonhierarchical Procedures 

There are strong arguments for not fixing the number of clusters, K , in advance, 
including the following: 

L If two or more seed points inadvertently lie within a single cluster, their resulting 
clusters will be poorly differentiated. 


Cluster profiles—variables are ordered by /•'-ratio size 
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2. The existence of an outlier might produce at least one group with very disperse 
items. 

3. Even if the population is known to consist of K groups, the sampling method 
may be such that data from the rarest group do not appear in the sample. Forc¬ 
ing the data into K groups would lead to nonsensical clusters. 

In cases in which a single run of the algorithm requires the user to specify K , it 
is always a good idea to rerun the algorithm for several choices. 

Discussions of other nonhierarchical clustering procedures are available in [3], 
[8], and [16]. 


12.5 Clustering Based on Statistical Models 

The popular clustering methods discussed earlier in this chapter, including single 
linkage, complete linkage, average linkage. Ward’s method and /C-means cluster¬ 
ing, are intuitively reasonable procedures but that is as much as we can say with¬ 
out having a model to explain how the observations were produced. Major 
advances in clustering methods have been made through the introduction of sta¬ 
tistical models that indicate how the collection of (p X 1) measurements x y , from 
the N objects, was generated. The most common model is one where cluster k has 
expected proportion p k of the objects and the corresponding measurements are 
generated by a probability density function /*(x). Then, if there are K clusters, the 
observation vector for a single object is modeled as arising from the mixing distri¬ 
bution 


K 

fhOxi*) = ^Pkfki*) 
k~l 

where each p k ^ 0 and P* = 1. This distribution / Mix (x) is called a mixture of 

the K distributions /i(x),/ 2 (x),...,/^(x) because the observation is generated 
from the component distribution f k (x) with probability p k . The collection of N ob¬ 
servation vectors generated from this distribution will be a mixture of observations 
from the component distributions. 

The most common mixture model is a mixture of multivariate normal distribu¬ 
tions where the A>th component f k (x) is the N p (p. ky X k ) density function. 

The normal mixture model for one observation x is 

fhtix(x I 


- 1 t'w«ix,iw“ p B<* ■ ■ '“>) <12 ‘ 17) 

Clusters generated by this model are ellipsoidal in shape with the heaviest concen¬ 
tration of observations near the center. 
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Inferences are based on the likelihood, which for N objects and a fixed number 
of clusters K , is 


£(/h> • ■ • 7 Pk, Mi> 2i,... , M*> ^a:) - IT fMix( x j I Mi> 2i,..., Mat* 2*) 

7-1 


= n ^ (27r)p/2 | IJ1/2 ex P ^-2 (X > - - #**)JJ (12-18) 

where the proportions p u ..., p k , the mean vectors /x 1? ... ,/i*, and the covariance 
matrices 2i,..., 2* are unknown. The measurements for different objects are 
treated as independent and identically distributed observations from the mixture 
distribution. 

There are typically far too many unknown parameters for parameters for mak¬ 
ing inferences when the number of objects to be clustered is at least moderate. 
However, certain conclusions can be made regarding situations where a heuristic 
clustering method should work well. In particular, the likelihood based procedure 
under the normal mixture model with all 2* the same multiple of the identity 
matrix, 171, is approximately the same as -means clustering and Ward’s method. 
To date, no statistical models have been advanced for which the cluster formation 
procedure is approximately the same as single linkage, complete linkage or average 
linkage. 

Most importantly, under the sequence of mixture models (12-17) for different 
K y the problems of choosing the number of clusters and choosing an appropriate 
clustering method has been reduced to the problem of selecting an appropriate sta¬ 
tistical model. This is a major advance. 

A good approach to selecting a model is to first obtain the maximum likelihood 
estimates p i9 ... , p K , /ii,2i f ...,Mjr, 2* for a fixed number of clusters K. These es¬ 
timates must be obtained numerically using special purpose software. The resulting 
value of the maximum of the likelihood 

^max £(£b • • * » PK* Mb 2j, . . . , p K y 1 K ) 

provides the basis for model selection. How do we decide on a reasonable value for 
the number of clusters K ? In order to compare models with different numbers 
of parameters, a penalty is subtracted from twice the maximized value of the 
log-likelihood to give 

-2 In L max - Penalty 

where the penalty depends on the number of parameters estimated and the number 
of observations N. Since the probabilities p k sum to 1, there are only K - 1 proba¬ 
bilities that must be estimated, AT x p means and K X p[p + l)/2 variances and 
covariances. For the Akaike information criterion (AIC), the penalty is 
2N X (number of parameters) so 


(12-19) 


AIC = 2 In L max -~2N\K-{p + 1 )(p + 2) - 1 
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The Bayesian information criterion (BIC) is similar but uses the logarithm of the 
number of parameters in the penalty function 

BIC = 2ln L max - 2\n(N)(Kj(p + 1 ){p + 2) - 1 j (12-20) 

There is still occasional difficulty with too many parameters in the mixture model so 
simple structures are assumed for the 2*. In particular, progressively more compli¬ 
cated structures are allowed as indicated in the following table. 

Assumed form Total number 

for 2* of parameters BIC 

2* = 7? I K(p + 1) lnL max - 2\n(N)K(p + 1) 

2* = Vk I K{p + 2) - 1 In L mm - 2\n{N){K(p + 2) - 1) 

2* = 7) k Diag(\i,\ 2 ,...,\ P ) K(p + 2) + p - 1 In L m , x - 2\n(N)(K(p + 2) + p — 1) 

Additional structures for the covariance matrices are considered in [6] and [9]. 

Even for a fixed number of clusters, the estimation of a mixture model is 
complicated. One current software package, MCLUST, available in the R software 
library, combines hierarchical clustering, the EM algorithm and the BIC criterion to 
develop an appropriate model for clustering. In the £ E’-step of the EM algorithm, a 
(N X K) matrix is created whose ;th row contains estimates of the conditional (on 
the current parameter estimates) probabilities that observation x ; belongs to cluster 
1,2 ,..., K. So, at convergence, the;th observation (object) is assigned to the cluster 
k for which the conditional probability 

p(k\xj) = p j f{Xj\k)l'2 l p i f{x i \k) 

1 = 1 

of membership is the largest. (See [6] and [9] and the references therein.) 

Example 12.13 (A model based clustering of the iris data) Consider the Iris data in 
Table 11.5. Using MCLUST and specifically the me function, we first fit the p = 4 
dimensional normal mixture model restricting the covariance matrices to satisfy 
= Vk 1 - * = 1 . 2 , 3 . 

Using the BIC criterion, the software chooses K = 3 clusters with estimated 
centers 



and estimated variance-covariance scale factors 7^ = .076, t) 2 = .163 and t) 3 = .163. 
The estimated mixing proportions are p x = .3333, p 2 = .4133 and p 3 = .2534. For 
this solution, BIC = -853.8. A matrix plot of the clusters for pairs of variables is 
shown in Figure 12.13. 

Once we have an estimated mixture model, a new object x ; - will be assigned to the 
cluster for which the conditional probability of membership is the largest (see [9]). 

Assuming the 2* = 17*1 covariance structure and allowing up to AT =7 clus¬ 
ters, the BIC can be increased to BIC = -705.1. 
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Figure 12 J 3 Multiple scatter plots of K = 3 clusters for Iris data 


Finally, using the BIC criterion with up to K = 9 groups and several different 
covariance structures, the best choice is a two group mixture model with uncon¬ 
strained covariances. The estimated mixing probabilities are p x = .3333 and 
p 2 = .6667. The estimated group centers are 



"5.or 


"6.26" 


3.43 


2.87 

Ml = 

1.46 

i M2 = 

4.91 


0.25_ 


_1.68_ 


and the two estimated covariance matrices are 


”.1218 

.0972 

.0160 

.0101“ 


.4530 

.1209 

.4489 

.1655" 

.0972 

.1408 

.0115 

.0091 

v ~ 

.1209 

,1096 

.1414 

.0792 

.0160 

.0115 

.0296 

.0059 

z 2 - 

.4489 

.1414 

.6748 

.2858 

_.0101 

.0091 

.0059 

,0109_ 


.1655 

.0792 

.2858 

.1786_ 



Essentially, two species of Iris have been put in the same cluster as the projected 
view of the scatter plot of the sepal measurements in Figure 12.14 shows. ■ 

12.6 Multidimensional Scaling 

This section begins a discussion of methods for displaying (transformed) multivari¬ 
ate data in low-dimensional space. We have already considered this issue wheil we 



Figure 12.14 Scatter plot of sepal measurements for best model. 


discussed plotting scores on, say, the first two principal components or the scores on 
the first two linear discriminants. The methods we are about to discuss differ from 
these procedures in the sense that their primary objective is to “fit” the original data 
into a low-dimensional coordinate system such that any distortion caused by a re¬ 
duction in dimensionality is minimized. Distortion generally refers to the similari¬ 
ties or dissimilarities (distances) among the original data points. Although 
Euclidean distance may be used to measure the closeness of points in the final low¬ 
dimensional configuration, the notion of similarity or dissimilarity depends upon 
the underlying technique for its definition. A low-dimensional plot of the kind we 
are alluding to is called an ordination of the data. 

Multidimensional scaling techniques deal with the following problem: For a set 
of observed similarities (or distances) between every pair of N items, find a repre¬ 
sentation of the items in few dimensions such that the interitem proximities “nearly 
match” the original similarities (or distances). 

It may not be possible to match exactly the ordering of the original similarities 
(distances). Consequently, scaling techniques attempt to find configurations in 
q < N - 1 dimensions such that the match is as close as possible. The numerical 
measure of closeness is called the stress . 

It is possible to arrange the N items in a low-dimensional coordinate system using 
only the rank orders of the N(N - 1 )/2 original similarities (distances), and not their 
magnitudes. When only this ordinal information is used to obtain a geometric repre¬ 
sentation, the process is called nonmetric multidimensional scaling. If the actual magni¬ 
tudes of the original similarities (distances) are used to obtain a geometric 
representation in q dimensions, the process is called metric multidimensional scaling . 
Metric multidimensional scaling is also known as principal coordinate analysis. 






708 Chapter 12 Clustering, Distance Methods, and Ordination 


workSsS h ng qU 2 e n%^ re d f Ve '°P ed by Shepard (see [29] for a review of early 
Kk i t 19 ' 20 : 21 ]. and others. A good summary of the history, theory and 
applications of multidimensional scaling is contained in [35], Multidimensional 
mg invaria y requires the use of a computer, and several good computer 

programs are now available for the purpose. B computer 

The Basic Algorithm 

For N items, there are M = N(N - l)/2 similarities (distances) between pairs o ‘ 
erent items. These similarities constitute the basic data. (In cases where the simi 
ors he / eaSdy k qUan * ified as - for exam Ple, the similarity between two co 

ors, the rank orders of the similarities are the basic data.) 

Assuming no ties, the similarities can be arranged in a strictly ascending order as 

„ . ^'i*i < s i 2 k 2 < < s, MkM (12-21) 

ere */,*, is the smallest of the M similarities. The subscript i 1 k l indicates the pai- 

dS: £f‘ *“'“*? Is. toe items -to r.ni 1 to the simt'X 

.. ®‘. . er su scn P ts are interpreted in the same manner. We want to find ”■ 

^-dimensional configuration of the N items such that the distances, d%\ between" 

LrTnnllT Til ° Fdering in < 12 ’ 21 )- If the distances are laid out in a manne- 
corresponding to that ordering, a perfect match occurs when 

d $i > 4*2 > > d\$ M (12-22; 

gone tn th CSCending ° rderin ® ° f tbe distances in q dimensions is exactly analo- 

12 T i n l aSCen Tl dng ° f the initial similarities. As long as the order in 
(12-22) is preserved, the magnitudes of the distances are unimportant. 

whTT glVen V f.‘ Ue ° f q ' 11 may not be P oss ible to find a configuration of points 

k a " "’°r onicai1 ' m 4™. 

tinn ln l 9 P r ° p0sed a mea sure of the extent to which a geometrical representa¬ 
tion falls short of a perfect match. This measure, the stress, is defined as 

[ 22 (^-#>)*)* 

Stress (q) = l ^*- 

( I 

TlT S , m . th ®, stre , ss formu l a are numbers known to satisfy (12-22); that is, they 

thatTeT aTfv T l h° **“ similarities - The ffl’s are ^/distances in the sense 
*TT y T T USUal dlStanCe Pr°P e rties of (1-25). They are merely reference 
numbers used to judge the nonmonotonicity of the observed 

that the Tessas arepresen T 0 " of the items as points in ^-dimensions such 
internr JIaT a- “ p0SS]ble - Kruskal [19] suggests the stress be informally 

interpreted according to the following guidelines: 


Stress 

Goodness of fit 

20% 

Poor 

10% 

Fair 

5% 

Good 

2.5% 

Excellent 

0% 

Perfect 


Sal d^stanws ; l ° rnonotonic relationship between the similarities and the 
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A second measure of discrepancy, introduced by Takane et al. is bec oming 
the preferred criterion. For a given dimension q, this measure, denoted by SStress, 
replaces the d tk ’s and d ik ’s in (12-23) by their squares and is given by 

22(<&-4) 2 T 2 

SStress = - (12-25) 

L i<k _ 

The value of SStress is always between 0 and 1. Any value less than .1 is typically 
taken to mean that there is a good representation of the objects by the points in the 
given configuration. 

Once items are located in q dimensions, theij q x 1 vectors of coordinates can be 
treated as multivariate observations. For display purposes, it is convenient to represent 
this ^-dimensional scatter plot in terms of its principal component axes. (See Chapter 8.) 

We have written the stress measure as a function of q , the number of dimensions 
for the geometrical representation. For each q , the configuration leading to the min¬ 
imum stress can be obtained. As q increases, minimum stress will, within rounding 
error, decrease and will be zero for q = N - 1. Beginning with q = 1, a plot of 
these stress (q) numbers versus q can be constructed. The value of q for which this 
plot begins to level off may be selected as the “best” choice of the dimensionality. 
That is, we look for an “elbow” in the stress-dimensionality plot. 

The entire multidimensional scaling algorithm is summarized in these steps: 

1. For TV items, obtain the M ~ N(N - l)/2 similarities (distances) between dis¬ 
tinct pairs of items. Order the similarities as in (12-21). (Distances are ordered 
from largest to smallest.) If similarities (distances) cannot be computed, the 
rank orders must be specified. 

2. Using a trial configuration in q dimensions, determiue the interitem distances d\ q k 
and numbers d $, where the latter satisfy (12-22) and minimize the stress (12-23) or 
SStress (12-25). (The d\f are frequently determined within scaling computer pro¬ 
grams using regression methods designed to produce monotonic “fitted” distances.) 

3. Using the s, move the points around to obtain an improved configuration. 
(For q fixed, an improved configuration is determined by a general function 
minimization procedure applied to the stress. In this context, the stress is re¬ 
garded as a function of the N Xq coordinates of the N items.) A new configu¬ 
ration will have new d\P's new d^' s and smaller stress. The process is repeated 
until the best (minimum stress) representation is obtained. 

4. Plot minimum stress (q) versus q and choose the best number of dimensions, q*> 

from an examination of this plot. (12-26) 

We have assumed that the initial similarity values are symmetric (s ik = s ki ) y that 
there are no ties, and that there are no missing observations. Kruskal [19, 20] has 
suggested methods for handling asymmetries, ties, and missing observations. In ad¬ 
dition, there are now multidimensional scaling computer programs that will handle 
not only Euclidean distance, but any distance of the Minkowski type. [See (12-3).] 
The next three examples illustrate multidimensional scaling with distances as 
the initial (dis)similarity measures. 

Example 12.14 (Multidimensional scaling of U.S. cities) Table 12.7 displays the 
airline distances between pairs of selected U.S. cities. 
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Figure 12.15 A geometrical representation of cities produced by multidimensional 
scaling. 


Since the cities naturally lie in a two-dimensional space (a nearly level part of the 
curved surface of the earth), it is not surprising that multidimensional scaling with 
q — 2 will locate these items about as they occur on a map. Note that if the distances 
in the table are ordered from largest to smallest—that is, from a least similar to most 
similar—the first position is occupied by ^Boston, l a. = 3052, 

A multidimensional scaling plot for q - 2 dimensions is shown in Figure 12.15. 
The axes lie along the sample principal components of the scatter plot. 

A plot of stress (q) versus q is shown in Figure 12.16 on page 712. Since 
stress (1) X 100% = 12%, a representation of the cities in one dimension (along a 
single axis) is not unreasonable. The “elbow” of the stress function occurs at q ~ 2. 
Here stress (2) X 100% = 0.8%, and the “fit” is almost perfect. 

The plot in Figure 12.16 indicates that q = 2 is the best choice for the dimen¬ 
sion of the final configuration. Note that the stress actually increases for q - 3. 
This anomaly can occur for extremely small values of stress because of difficulties 
with the numerical search procedure used to locate the minimum stress. ■ 

Example 12.15 (Multidimensional scaling of public utilities) Let us try to represent 
the 22 public utility firms discussed in Example 12.7 as points in a low-dimensional 
space. The measures of (dis)similarities between pairs of firms are the Euclidean 
distances listed in Table 12.6. Multidimensional scaling in q = 1, 2,..., 6 dimensions 
produced the stress function shown in Figure 12.17. 
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Figure 12.16 Stress function for airline distances between cities. 


Stress 



Figure 12.17 Stress function for distances between utilities. 
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Figure 12.18 A geometrical representation of utilities produced by multidimensional 
scaling. 


The stress function in Figure 12.17 has no sharp elbow. The plot appears to level 
out at “good” values of stress (less than or equal to 5%) in the neighborhood of 
q = 4. A good four-dimensional representation of the utilities is achievable, but dif¬ 
ficult to display. We show a plot of the utility configuration obtained in q - 2 di¬ 
mensions in Figure 12.18. The axes lie along the sample principal components of the 
final scatter. 

Although the stress for two dimensions is rather high (stress (2) X 100% = 
19%), the distances between firms in Figure 12.18 are not wildly inconsistent with 
the clustering results presented earlier in this chapter. For example, the midwest 
utilities—Commonwealth Edison, Wisconsin Electric Power (WEPCO), Madison 
Gas and Electric (MG & E), and Northern States Power (NSP)—are close together 
(similar). Texas Utilities and Oklahoma Gas and Electric (Ok. G & E) are also very 
close together (similar). Other utilities tend to group according to geographical 
locations or similar environments. 

The utilities cannot be positioned in two dimensions such that the interutility 
distances d$ are entirely consistent with the original distances in Table 12.6. More 
flexibility for positioning the points is required, and this can only be obtained by in¬ 
troducing additional dimensions. ■ 

Example 12.16 (Multidimensional scaling of universities) Data related to 25 U.S. 
universities are given in Table 12.9 on page 729. (See Example 12.19.) These data 
give the average SAT score of entering freshmen, percent of freshmen in top 
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Figure 12.19 A two-dimensional representation of universities produced by metric 
multidimensional scaling. 


Figure 12.20 A two-dimensional representation of universities produced by nonmetric 
multidimensional scaling. 


10% of high school class, percent of applicants accepted, student-faculty ratio, esti¬ 
mated annual expense, and graduation rate (%). A metric multidimensional scaling 
algorithm applied to the standardized university data gives the two-dimensional 
representation shown in Figure 12.19. Notice how the private universities cluster 
on the right of the plot while the large public universities are, generally, on the left. 
A nonmetric multidimensional scaling two-dimensional configuration is shown in 
Figure 12.20. For this example, the metric and nonmetric scaling representations 
are very similar, with the two dimensional stress value being approximately 10% 
for both scalings. * a 


To summarize, the key objective of multidimensional scaling procedures is a 
low-dimensional picture. Whenever multivariate data can be presented graphically 
in two or three dimensions, visual inspection can greatly aid interpretations. 

When the multivariate observations are naturally numerical, and Euclidean dis¬ 
tances in p-dimensions, dj£\ can be computed, we can seek a q < / 7 -dimensional 
representation by minimizing 


E = 


22 (<&> - d^f/W 


22 ^T 1 

_ i<k 


(12-27) 


Classical metric scaling, or principal coordinate analysis,is equivalent to ploting 
the principal components. Different software programs choose the signs of the ap¬ 
propriate eigenvectors differently, so at first sight, two solutions may appear to be 
different. However, the solutions will coincide with a reflection of one or more of 
the axes. (See [26].) 


In this alternative approach, the Euclidean distances in p and q dimensions are 
compared directly. Techniques for obtaining low-dimensional representations by 
minimizing E are called nonlinear mappings. 

The final goodness of fit of any low-dimensional representation can be 
depicted graphically by minimal spanning trees . (See [16] for a further discussion of 
these topics.) 
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12.7 Correspondence Analysis 

Developed by the French, correspondence analysis is a graphical procedure for rep¬ 
resenting associations in a table of frequencies or counts. We will concentrate on a 
two-way table of frequencies or contingency table. If the contingency table has I 
rows and J columns, the plot produced by correspondence analysis contains two sets 
of points: A set of / points corresponding to the rows and a set of J points corre¬ 
sponding to the columns. The positions of the points reflect associations. 

Row points that are close together indicate rows that have similar profiles (con¬ 
ditional distributions) across the columns. Column points that are close together in¬ 
dicate columns with similar profiles (conditional distributions) down the rows. 
Finally, row points that are close to column points represent combinations that 
occur more frequently than would be expected from an independence model—that 
is, a model in which the row categories are unrelated to the column categories. 

The usual output from a correspondence analysis includes the “best” two- 
dimensional representation of the data, along with the coordinates of the plotted 
points, and a measure (called the inertia) of the amount of information retained in 
each dimension. 

Before briefly discussing the algebraic development of correspondence analy¬ 
sis, it is helpful to illustrate the ideas we have introduced with an example. 

Example 12.17 (Correspondence analysis of archaeological data) Table 12.8 contains 
the frequencies (counts) of / = 4 different types of pottery (called potsherds) 
found at J = 7 archaeological sites in an area of the American Southwest. If we 
divide the frequencies in each row (archaeological site) by the corresponding row 
total, we obtain a profile of types of pottery. The profiles for the different sites 
(rows) are shown in a bar graph in Figure 12.21(a). The widths of the bars are 
proportional to the total row frequencies. In general, the profiles are different; 
however, the profiles for sites PI and P2 are similar, as are the profiles for sites 
P4 and P5. 

The archaeological site profile for different types of pottery (columns) are 
shown in a bar graph in Figure 12.21(b). The site profiles are constructed using the 

Table 12.8 Frequencies of Types of Pottery 


Site 

Type 

Total 

A 

B 

C 

D 

PO 

30 

10 

10 

39 

89 

PI 

53 

4 

16 

2 

75 

P2 

73 

1 

41 

1 

116 

P3 

20 

6 

1 

4 

31 

P4 

46 

36 

37 

13 

132 

P5 

45 

6 

59 

10 

120 

P6 

16 

28 

169 

5 

218 

Total 

283 

91 

333 

74 

781 


Source: Data courtesy of M. J. Tretter. 


Type 
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Figure 12.21 Site and pottery type profiles for the data in Table 12.8. 


column totals. The bars in the figure appear to be quite different from one another. 
This suggests that the various types of pottery are not distributed over the archaeo¬ 
logical sites in the same way. 

The two-dimensional plot from a correspondence analysis 2 of the pottery 
type-site data is shown in Figure 12.22. 

The plot in Figure 12.22 indicates, for example, that sites PI and P2 have similar 
pottery type profiles (the two points are close together), and sites PO and P6 have very 
different profiles (the points are far apart). The individual points representing the 
types of pottery are spread out, indicating that their archaeological site profiles are 
quite different. These findings are consistent with the profiles pictured in Figure 12.21. 

Notice that the points PO and D are quite close together and separated from the 
remaining points. This indicates that pottery type D tends to be associated, almost 
exclusively, with site PO. Similarly, pottery type A tends to be associated with site PI 
and, to lesser degrees, with sites P2 and P3. Pottery type B is associated with sites P4 
and P5, and pottery type C tends to be associated, again, almost exclusively, with site 
P6. Since the archaeological sites represent different periods, these associations are 
of considerable interest to archaeologists. 

The number A 2 = .28 at the end of the first coordinate axis in the two- 
dimensional plot is the inertia associated with the first dimension. This inertia is 55% 
of the total inertia. The inertia associated with the second dimension is A 2 = .17, and 
the second dimension accounts for 33% of the total inertia. Together, the two di¬ 
mensions account for 55% + 33% = 88% of the total inertia. Since, in this case, the 
data could be exactly represented in three dimensions, relatively little information 
(variation) is lost by representing the data in the two-dimensional plot of 
Figure 12.22. Equivalently, we may regard this plot as the best two-dimensional rep¬ 
resentation of the multidimensional scatter of row points and the multidimensional 


2 The JMP software was used for a correspondence analysis of the data in Table 12.8. 
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Figure 12.22 A correspondence analysis plot of the pottery type-site data. 


scatter of column points. The combined inertia of 88% suggests that the representa¬ 
tion “fits” the data well. 

In this example, the graphical output from a correspondence analysis shows the 
nature of the associations in the contingency table quite clearly. ■ 


Algebraic Development of Correspondence Analysis 

To begin, let X, with elements x ij9 be an 7 X 7 two-way table of unsealed fre¬ 
quencies or counts. In our discussion we take 7 > 7 and assume that X is of full 
column rank 7. The rows and columns of the contingency table X correspond to 
different categories of two different characteristics. As an example, the array of 
frequencies of different pottery types at different archaeological sites shown in 
Table 12.8 is a contingency table with 7 = 7 archaeological sites and 7 = 4 pot¬ 
tery types. 

If n is the total of the frequencies in the data matrix X, we first construct a ma¬ 
trix of proportions P = { A;} by dividing each element of X by n. Hence 



i = 1 , 2 ,...,/, 


j = 1,2,..., 7, or 


P 

(IXJ) 


- X (12-28) 
n (Ixj) 


The matrix P is called the correspondence matrix . 
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Next define the vectors of row and column sums r and c respectively, and the 
diagonal matrices D r and D c with the elements of r and c on the diagonals. Thus 


r i = 2 Pij = 2 ~zr> i = 1,2,...,/, or r = P 1, 

i= 1 ;«1 n (/xl) (/x/)(7xl) 

1 tA x ij 

c j = 2 Pij ~ 2 7 = 1,2,...,/, or c = P' 1/ 

" ,=i n ( yxi) (/x/)(/xi) 


(12-29) 


where 1 7 is a 7 X 1 and 1/ is a / X 1 vector of l’s and 

D, = diag ( r x , r 2 ,..., r,) and D c = diag (cj, c 2 ,..., cj) (12-30) 
We define the square root matrices 

D) /2 = diag(V^,..., Vn) D7 I/2 = diag^A=,..., 

, v (12-31) 

D 1 / 2 = diag(Vci,..., VcJ) D7 1/2 = diagf A=,..., 

for scaling purposes. 

Correspondence analysis can be formulated as the weighted least squares prob¬ 
lem to select P = {pij}, a matrix of specified reduced rank, to minimize 

2 2 (Pi ' ~~ = tr[(D7 1/2 (P - P)P 7 1 / 2 ) (D 7 ^( P _ P)D7^)'] (12-32) 

1 = 1 j~\ r i C ) 

since (p,- ; — Pij)/Vr^Cj is the (i,j) element of DJT^P ~ P)D^ 2 . 

As Result 12.1 demonstrates, the term re' is common to the approximation P 
whatever the 7 X 7 correspondence matrix P.The matrix P = rc' can be shown to 
be the best rank 1 approximation to P. 

Result 12.1. The term rc' is common to the approximation P whatever the 7 x 7 
correspondence matrix P. 

The reduced rank s approximation to P, which minimizes the sum of squares 
(12-32), is given by 

p = 2 M°r 1/2 ■*) (°c /2 v*)' = rc' + 2 A*(D^ n*) (D^ v*)' 

k = l k-2 

where the X k are the singular values and the 7x1 vectors u* and the 7X1 vectors 
\ k are the corresponding singular vectors of the 7 X 7 matrix D^PDJ 1 / 2 . The 

minimum value of (12-32) is ^ A*. 

k~s +1 

The reduced rank K > 1 approximation to P - rc' is 


P-rc'= 2 3*(DJ /2 u*)(Dy 2 v*)' 


(12-33) 
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where the A* are the singular values and the IX 1 vectors u k and the 7X1 vectors 
y k are the corresponding singular vectors of the I X J matrix D7 1/2 (P - rc')D7 1/2 . 
Here X k = A* +1 , u k =.u* +1 , and \ k — for k = 1,..., J - 1. 

Proof. We first consider a sealed version B = D7 1/2 PD7 1/2 of the correspondence 
matrix P. According to Result 2A.16, the best low rank = ^ approximation B to 
d^pd: 1 / 2 is given by the first s terms in the the singular-value decomposition 

D^PD^ = 2 A*n t Tl (12-34) 

k=l 

where 

D-iApo-i/ 2 ^ = A*u* uiD-^PDJ 1 ^ = A*vi (12-35)' 

and 

| (D7 1/2 PD" 1/2 ) (D7 1/Z PD - A*I| = 0 for k = l,...,J 
The approximation to P is then given by 

p = Dy 2 BDy 2 = s a,(d;/ 2 u,)(d pv k y 

k—1 

J ~ 

and, by Result 2A.16, the error of approximation is 2 A*. 

k=s +1 

Whatever the correspondence matrix P, the term re' always provides a (the 
best) rank one approximation. This corresponds to the assumption of independence 
of the rows and columns. To see this, let Uj = Dj/ 2 !/ and = D^ly, where 1; is a 
7X1 and lj a J X 1 vector of l’s. We verify that (12-35) holds for these choices. 

u',(D 7 1/2 PD7 1/2 ) = (D^l/HD^PD^) 

- = l^PD^ 2 = c'D7 1/2 
= [Vq,..., Vcj] = (Dj^ly)' = vi 

and 

(Dr^PDr^V! = (D7 1 /2 PD -i/2)( D i/2 ly ) 

= D7^Ply = Dr^r 

rv^i 

= : = D^i/ = s! 

Lv^J 

That is, 

= (D^.DWy (12-36) 

are singular vectors associated with singular value A a = 1. For any correspondence 
matrix, P, the common term in every expansion is 

D^UjViD'/ 2 = D r l/1)D C = rc' 


! 
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Therefore, we have established the first approximation and (12-34) can always be 
expressed as 

P = rc' + 2 A,(D‘/ 2 u,)(Dj/ 2 v,)’ 

k=2 

Because of the common term, the problem can be rephrased in terms of P - rc' 
and its scaled version D r 1 ^ 2 (P — rc^D^ 1 ^ 2 . By the orthogonality of the singular 
vectors of D; 1/2 PD c _1/2 , we have o^D^ 2 !,) = 0 and \' k (T> l J 2 lj) = 0, for k > 1, so 

D7 1/2 (P - rc')V-W = 2 A 

k=2 

is the singular-value decomposition of D~ 1/2 (P - rc' ) D~ 1/2 in terms of the singular val¬ 
ues and vectors obtained from D7^ 2 PD7 1/7 . Converting to singular values and vectors 
A*, u*, and v* from D7 ! ^(P — rc')D7 1//2 only amounts to changing k to k — 1 so 
h = A* +1 ,u* = u* +1 ,andv* = v k+l foTk = 1,...,/ - 1. 

In terms of the singular value decomposition for D7 1/2 (P — rc')D7 1/2 , the ex¬ 
pansion for P — rc' takes the form 

P - rc' = 2 A*(D; /2 u*) (D^v*)' ( 12 - 37 ) 

k = l 

K 

The best rank K approximation to D7 1/2 (P - rc')D7 1/2 is given by 2 A*u*v*. 
Then, the best approximation to P — rc' is k=1 

P - rc = 2 A*(D^u*) (Dj/ 2 v*)' (12-38) 

k=\ 


Remark. Note that the vectors D l J 2 u k and in the expansion (12-38) of 

P - rc' need not have length 1 but satisfy the scaling 

(DyWD^D^-u*) = u^u* = 1 
(D^nJ'D-HDV***) = v^v* = 1 

Because of this scaling, the expansions in Result 12.1 have been called a generalized 
singular-value decomposition. 

Let A, U = [u lt ..., u/] and V = [\ u ..., \j] be the matricies of singular values 
and vectors obtained from D7^ 2 (P — rc')D7^ 2 . It is usual in correspondence 
analysis to plot the first two or three columns of F = D^Dj^U) A and 
G = D7 J (DpV) A or A*D;T 1/2 u* and X k D~^ 2 \ k for k = 1, 2, and maybe 3. 

The joint plot of the coordinates in F and G is called a symmetric map (see 
Greenacre [13]) since the points representing the rows and columns have the same 
normalization, or scaling, along the dimensions of the solution. That is, the geometry 
for the row points is identical to the geometry for the column points. 
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Example 12.18 (Calculations for correspondence analysis) Consider the 3X2 
contingency table 



B1 

B2 

Total 

A 1 

24 

12 

36 

A 2 

16 

48 

64 

A 3 

60 

40 

100 


100 

100 

200 


The correspondence matrix is 


P = 


.12 

.08 

.30 


.06 

.24 

.20 


with marginal totals c' = [.5, .5] and r' = [.18, .32, .50]. The negative square root 
matrices are 


> 


1 

> 

> 

T 

A 

> 

> 

% 

> 

% 

% 


D; 1/2 = diag ( V2/.6, V2/.8, V2) D' 1 / 2 = diag(\^, V2) 


Then 


.12 .06 


~.18~ 


.03 -.03 

<N 

00 

O 

- 

.32 

[.5 .5] = 

-.08 .08 

_.30 .20, 


_.50_ 


o 

f 

p 


The scaled version of this matrix is 


A = D; 1 / 2 ^ - rc') D' 1 / 2 - 


V2 

.6 

0 

0 


0 

V2 

.8 

0 


.03 

-. 03 “ 

~V2 

o" 

-.08 

.08 

0 

V2_ 

.05 

-.05 _ 




V2 


0.1 

- 0.1 

- 0.2 

0.2 

0.1 

- 0.1 


Since I > the square of the singular values and the v ; - are determined from 




.1 

—. l " 




.1 -.2 

.1 

.2 

-. 1 _ 


.06 

-.06 

-.1 .2 

-. 1 _ 

-.2 

.1 


-.06 

.06 _ 


1 


i 

l 

l 

j 


f 

i 


It is easily checked that A? = .12, A^ = 0, since / - 1 = 1, and that 

r 11 



V2 


Further, 


AA' = 


.1 

-.2 

-.r 

.2 

.1 

-.2 

; 1 . 

1 

i 

© © 

-.04 

.08 

. 02 “ 

-.04 

.1 

-. 1 _ 

-.1 

.2 

-•ij 

.02 

-.04 

.02 _ 


A computer calculation confirms that the single nonzero eigenvalue is A 2 = .12, 
so that the singular value has absolute value A L = .2V3 and, as you can easily 
check, 


Ui = 


1 

V6 

2 

v! 

i 

V6 


The expansion of P — rc' is then the single term 

A 1 (Dy2 Ul )( D y2 Vl) ' 


= VT2 


.6 

v5 

0 

0 


0 

.8 

v5 

0 


0 

0 

1 

V2 



i' 


V6 


2 


V6 


1 


V6_ 


1 -1 

V2 V5 


1 

V5 

0 


0 

1 

V2 


= VT2 


.3 

.8 

V3 

.5 

v5 


1 

-ii 

.03 

-.03“ 

2 

2j 

-.08 

.08 



.05 

-.05 _ 


check 


A'A = 





\ 

\ 

N 

\ 

S 

\ 
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There is only one pair of vectors to plot 

0 

I V2 

AjD^Uj = VJ2 


.8 

V2 


0 


V2 



1 ■ 


.3 " 


V6 


V3 


2 

V6 

= Vl2 

.8 

v5 


1 


.5 


V6_ 


V3 


and 



1 

0 

1 


■ 1 ' 


r 

, AjD^Vj = Vl2 

V2 

0 


V2 

-i 

= V7l2 

2 

1 






2_ 


There is a second way to define contingency analysis. Following Greenacre [13], 
we call the preceding approach the matrix approximation method and the approach 
to follow the profile approximation method. We illustrate the profile approximation 
method using the row profiles; however, an analogous solution results if we were to 
begin with the column profiles. 

Algebraically, the row profiles are the rows of the matrix D7 J P, and contin¬ 
gency analysis can be defined as the approximation of the row profiles by points in 
a low-dimensional space. Consider approximating the row profiles by the matrix P*. 
Using the square-root matrices D^ 2 and Dj /2 defined in (12-31), we can write 

(d^p - p*)d^ 2 = d; 1 / 2 (d; 1/2 p - dJ^p*)©^ 

and the least squares criterion (12-32) can be written, with p*j = Pij/ r h as 

2^ z, ——-= 2 n 2--- 

i / r i C ) i j C j 

= tr [Dy 2 Dj/ 2 (p7 ] P - P*)D7 1/2 D7 1/2 (D; 1 P - P*)'] 

= tr[Dj/ 2 (D; 1/2 P - Dy 2 P*)D7 1 ' /2 D7 1/2 (D7 1/2 P - Dj^P*)'D7 1/2 ] 

= tr[[(D^P - Dj /2 P*)D7 1/2 ][(D; 1/2 P - D; /2 P*)D7 1/2 ]'] (12-39) 

Minimizing the last expression for the trace in (12-39) is precisely the first min¬ 
imization problem treated in the proof of Result 12.1. By (12-34), D7 1/2 PD7 1/2 has 
the singular-value decomposition 

D7 1/2 PD~ 1/2 = 2 J k u k y‘ k (12-40) 

k = 1 

The best rank K approximation is obtained by using the first K terms of this expan¬ 
sion. Since, by (12-39), we have D7 ly/2 PD*/ 2 approximated by Dj^P^DJ 1 ^ 2 , we left 
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multiply by D7 1/2 and right multiply by D l J 2 to obtain the generalized singular-value 
decomposition 

D; l P = 2 A*D7 I/2 u*(Dy 2 v* )' (12-41) 

where, from (12-36), (u!, Vj) = (Dj^l./, Dj /2 1/) are singular vectors associated with 
singular value ^ = 1. Since D7 1/2 (D r 1/2 l 7 ) = 1 7 and (Dj^l ~ c', the leading 
term in the decomposition (12-41) is l 7 c'. 

Consequently, in terms of the singular values and vectors from D7 1/2 PD7 1/2 , the 
reduced rank K < J approximation to the row profiles D7*P is 

P* = l,c' + 2 A*D7 I/2 n*(Dj/ 2 ?*)' (12-42) 

k~2 

In terms of the singular values and vectors A*,u* and \ k obtained from 
D7 1/? (P - rc') DJ 1/2 , we can write 

P* - l,c' 2 A*D7 I/2 u*(Dj /2 v*)' 

k =1 

(Row profiles for the archaeological data in Table 12.8 are shown in Figure 12.21 on 
page 717.) 


Inertia 


Total inertia is a measure of the variation in the count data and is defined as the 
weighted sum of squares 


tr[D7 1/2 (P - rc')D c H / 2 (D7 1 /2(P - rc')D c ^ 2 )'] 


22 


(p.7 - r ffy_ 

nc. 


= 2 

(12-43) 


where the A* are the singular values obtained from the singular-value decomposi¬ 
tion of D7 1/2 (P - rc') D7 1/2 (see the proof of Result 12.1). 3 

The inertia associated with the best reduced rank K < J approximation to the 

centered matrix P — rc' (the /^-dimensional solution) has inertia 2 Th e 

k = l 

residual inertia (variation) not accounted for by the rank K solution is equal to the 
sum of squares of the remaining singular values: A^+i + A+ • • • + A 2 -!. For 
plots, the inertia associated with dimension k, A*, is ordinarily displayed along the 
kth coordinate axis, as in Figure 12.22 for k = 1,2. 




3 Total inertia is related to the chi-square measure of association in a two-way contingency table, 
(O t -i ~ Erf 




• Here O tj = x if is the observed frequency and E ti is the expected frequency for 


the ijth cell. In our context, if the row variable is independent of (unrelated to) the column variable, 
Eij = n r t c n and 

' 4. (Pa ~ 'icif x 2 

Total inertia ~ Z Z ~ - 


.-l /-i 


ncj 
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Interpretation in Two Dimensions 

Since the inertia is a measure of the data table's total variation, how do we interpret 
a large value for the proportion (A? + A 2 )/2 Geometrically, we say that the 

k=l 

associations in the centered data are well represented by points in a plane, and this 
best approximating plane accounts for nearly all the variation in the data beyond 
that accounted for by the rank 1 solution (independence model). Algebraically, we 
say that the approximation 

P - rc' = AjUiVj + A 2 u 2 v 2 
is very good or, equivalently, that 

P = rc' + AiU^i -I- A 2 u 2 v 2 

Final Comments 

Correspondence analysis is primarily a graphical technique designed to represent 
associations in a low-dimensional space. It can be regarded as a scaling method, and 
can be viewed as a complement to other methods such as multidimensional scaling 
(Section 12.6) and biplots (Section 12.8). Correspondence analysis also has links to 
principal component analysis (Chapter 8) and canonical correlation analysis 
(Chapter 10). The book by Greenacre [14] is one choice for learning more about 
correspondence analysis. 


12.8 Biplots for Viewing Sampling Units and Variables 

A biplot is a graphical representation of the information in an n x p data matrix. 
The bi- refers to the two kinds of information contained in a data matrix. The infor¬ 
mation in the rows pertains to samples or sampling units and that in the columns 
pertains to variables. 

When there are only two variables, scatter plots can represent the information 
on both the sampling units and the variables in a single diagram.This permits the vi¬ 
sual inspection of the position of one sampling unit relative to another and the rela¬ 
tive importance of each of the two variables to the position of any unit. 

With several variables, one can construct a matrix array of scatter plots, 
but there is no one single plot of the sampling units. On the other hand, a two- 
dimensional plot of the sampling units can be obtained by graphing the first two 
principal components, as in Section 8.4. The idea behind biplots is to add the infor¬ 
mation about the variables to the principal component graph. 

Figure 12.23 gives an example of a biplot for the public utilities data in 
Table 12.4. 

You can see how the companies group together and which variables con¬ 
tribute to their positioning within this representation. For instance, X 4 — annual 
load factor and Xg = total fuel costs are primarily responsible for the grouping of 
the mostly coastal companies in the lower right. The two variables X 1 = fixed- 



charge ratio and Y 2 — rate of return on capital put the Florida and Louisiana 
companies together. 

Constructing Biplots 

The construction of a biplot proceeds from the sample principal components. 

According to Result 8A.1, the best two-dimensional approximation to the data 
matrix X approximates the yth observation x ; in terms of the sample values of the 
first two principal components. In particular, 

*/ - X + yjl*l + yj2*2 (12-44) 

where ej and e 2 are the first two eigenvectors of S or, equivalently, of 
= ( n ~~ l)s. Here X c denotes the mean corrected data matrix with rows 
(x ; — x) . The eigenvectors determine a plane, and the coordinates of the yth unit 
(row) are the pair of values of the principal components, (y ; i, y ;2 ). 

To include the information on the variables in this plot, we consider the pair of 
eigenvectors (ej, e 2 ). These eigenvectors are the coefficient vectors for the first two 
sample principal components. Consequently, each row of the matrix E = [e^ e 2 ] 
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positions a variable in the graph, and the magnitudes of the coefficients (the coordi¬ 
nates of the variable) show the weightings that variable has in each principal com¬ 
ponent. The positions of the variables in the plot are indicated by a vector. Usually, 
statistical computer programs include a multiplier so that the lengths of all of the 
vectors can be suitably adjusted and plotted on the same axes as the sampling units 
Units that are close to a variable likely have high values on that variable. To inter¬ 
pret a new point x 0 , we plot its principal components E'(x 0 - x). 

A direct approach to obtaining a biplot starts from the singular value decom¬ 
position (see Result 2A.15), which first expresses the n X p mean corrected 
matrix X c as 

X c = U A V (12-45). 

(nXp) ( n *P) (P*P) ( P X P ) 

where A = diag (Aj, A 2> ..«, A^,) and V is an orthogonal matrix whose columns are the 
eigenvectors of X' X c = (n - 1)S. That is, V = E = [e x , e 2 , -. -, i p ]- Multiplying 
(12-45) on the right by E, we find 

X C E = UA (12-46) 

where the ;th row of the left-hand side, 

[ (Xy - i) f e lf (x ; - x)'e 2 ,-.., (x y - x)'e,] = [y jU ■ • > >/#J 

is just the value of the principal components for the ;th item. That is, UA contains all 
of the values of the principal components, while V = E contains the coefficients 
that define the principal components. 

The best rank 2 approximation to X c is obtained by replacing A by 
A* = diag(Ai, A 2 ,0,..., 0). This result, called the Eckart-Young theorem, was es¬ 
tablished in Result 8.A.l.The approximation is then 

X C *UA*V' = [*,*][!] d 2 - 47 ) 

where y x is the n X 1 vector of values of the first principal component and y 2 is the 
n X 1 vector of values of the second principal component. 

In the biplot, each row of the data matrix, or item, is represented by the point lo¬ 
cated by the pair of values of the principal components. The zth column of the data 
matrix, or variable, is represented as an arrow from the origin to the point with co¬ 
ordinates (e Ui e 2i ), the entries in the zth column of the second matrix [e 1} e 2 ]' in the 
approximation (12-47). This scale may not be compatible with that of the principal 
components, so an arbitrary multiplier can be introduced that adjusts all of the vec¬ 
tors by the same amount. 

The idea of a biplot, to represent both units and variables in the same plot, ex¬ 
tends to canonical correlation analysis, multidimensional scaling, and even more 
complicated nonlinear techniques. (See [12].) 


Example 12.19 (A biplot of universities and their characteristics) Table 12.9 gives the 
data on some universities for certain variables used to compare or rank major 
universities. These variables include X\ = average SAT score of new freshmen, 
X 2 — percentage of new freshmen in top 10% of high school class, X 3 = percentage 
of applicants accepted, X 4 = student-faculty ratio, X 5 — estimated annual expens¬ 
es and X$ = graduation rate (%). 

Because two of the variables, SAT and Expenses, are on a much different scale 
from that of the other variables, we standardize the data and base our biplot on the 
matrix of standardized observations z r The biplot is given in Figure 12.24 on 
page 730. 

Notice how Cal Tech and Johns Hopkins are off by themselves; the variable 
Expense is mostly responsible for this positioning. The large state universities in our 
sample are to the left in the biplot, and most of the private schools are on the right. 


Table 12.9 Data on Universities 

University 

SAT 

Top 10 

Accept 

SFRatio 

Expenses 

Grad 

Harvard 

14.00 

91 

14 

11 

39.525 

97 

Princeton 

13.75 

91 

14 

8 

30.220 

95 

Yale 

13.75 

95 

19 

11 

43.514 

96 

Stanford 

13.60 

90 

20 

12 

36.450 

93 

MIT 

13.80 

94 

30 

10 

34.870 

91 

Duke 

13.15 

90 

30 

12 

31.585 

95 

CalTech 

14.15 

100 

25 

6 

63.575 

81 

Dartmouth 

13.40 

89 

23 

10 

32.162 

95 

Brown 

13.10 

89 

22 

13 

22.704 

94 

JohnsHopkins 

13.05 

75 

44 

7 

58.691 

87 

UChicago 

12.90 

75 

50 

13 

38.380 

87 

UPenn 

12.85 

80 

36 

11 

27.553 

90 

Cornell 

12.80 

83 

33 

13 

21.864 

90 

Northwestern 

12.60 

85 

39 

11 

28.052 

89 

Columbia 

13.10 

76 

24 

12 

31.510 

88 

NotreDame 

12.55 

81 

42 

13 

15.122 

94 

UVirginia 

12.25 

77 

44 

14 

13.349 

92 

Georgetown 

12.55 

74 

24 

12 

20.126 

92 

CamegieMellon 

12.60 

62 

59 

9 

25.026 

72 

UMichigan 

11.80 

65 

68 

16 

15.470 

85 

UCBerkeley 

12.40 

95 

40 

17 

15.140 

78 

UWisconsin 

10.85 

40 

69 

15 

11.857 

71 

PennState 

10.81 

38 

54 

18 

10.185 

80 

Purdue 

10.05 

28 

90 

19 

9.066 

69 

TexasA&M 

10.75 

49. 

67 

25 

8.704 

67 

Source: US. News & World Report, September 18,1995,p. 126. 
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Figure 12.24 A biplot of the data on universities. 

Large values for the variables SAT,ToplO, and Grad are associated with the private 
school group. Northwestern lies in the middle of the biplot. ■ 

A newer version of the biplot, due to Gower and Hand [12], has some advan¬ 
tages. Their biplot, developed as an extension of the scatter plot, has features that 
make it easier to interpret. 

• The two axes for the principal components are suppressed. 

• An axis is constructed for each variable and a scale is attached. 

As in the original biplot, the i-th item is located by the corresponding pair of 
values of the first two principal components 

(Pi/, hi) = ((*; “ x)'ei,(*; - *)'e 2 ) 

where e x and where e 2 are the first two eigenvectors of S. The scales for the princi¬ 
pal components are not shown on the graph. 

In addition the arrows for the variables in the original biplot are replaced by 
axes that extend in both directions and that have scales attached. As was the case 
with the arrows, the axis for the i-the variable is determined by the i-the row of 
E = [«!, e 2 ]. 


To begin, we let Uj the vector with 1 in the i-th position and 0’s elsewhere. Then 
an arbitrary p X 1 vector x can be expressed as 

p 

x = 

i=i 

and, by Definition 2.A.12, its projection onto the space of the first two eigenvectors 
has coefficient vector 

E'x = i>,(E' Ui ) 

i=l 

so the contribution of the i-th variable to the vector sum is ^(E'u,) = X/[e lf , e^]'. 

The two entries e u and e 2 , in the i-the row of E determine the direction of the axis 
for the i-th variable. 

The projection vector of the sample mean x = 2f=i*« u * 

E'x= 2 *,(E'u,) 

1 = 1 

is the origin of the biplot. Every x can also be written as x = x + (x - x) and its 
projection vector has two components 

i;*,(E'o + £(x, - *,)(£'■,) 

i=l i=l 

Starting from the origin, the points in the direction w[eu, e 2l ]' are plotted for 
w = 0, ± 1, ± 2,... This provides a scale for the mean centered variable x t - x f . It 
defines the distance in the biplot for a change of one unit in x r But, the origin for 
the i-th variable corresponds to w = 0 because the term jc* ( E'u/) was ignored. 
The axis label needs to be translated so that the value x, is at the origin of the biplot. 
Since x, is typically not an integer (or another nice number), an integer (or other 
nice number) closest to it can be chosen and the scale translated appropriately. 
Computer software simplifies this somewhat difficult task. 

The scale allows us to visually interpolate the position of x t [e lh e ^]' in the 
biplot. The scales predict the values of a variable, not give its exact value, as they are 
based on a two dimensional approximation. 


Example 12.20 (An alternative biplot for the university data) We illustrate this 
newer biplot with the university data in Table 12.9. The alternative biplot with an 
axis for each variable is shown in Figure 12.25. Compared with Figure 12.24, the 
software reversed the direction of the first principal component. Notice, for exam¬ 
ple, that expenses and student faculty ratio separate Cal Tech and Johns Hopkins 
from the other universities. Expenses for CalTech and Johns Hopkins can be seen to 
be about 57 thousand a year, and the student faculty ratios are in the single digits. 
The large state universities, on the right hand side of the plot, have relatively high 
student faculty ratios, above 20, relatively low SAT scores of entering freshman, and 
only about 50% or fewer of their entering students in the top 10% of their high 
school class. The scaled axes on the newer biplot are more informative than the 
arrows in the original biplot. ■ 
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ToplO 


SAT 


Accept 

Expenses 

Figure 12.25 An alternative biplot of the data on universities. 

See le Roux and Gardner [23] for more examples of this alternative biplot and 
references to appropriate special purpose statistical software. 

12.9 Procrustes Analysis: A Method 
for Comparing Configurations 

Starting with a given n X n matrix of distances D, or similarities S, that relate n 
objects, two or more configurations can be obtained using different techniques.The 
possible methods include both metric and nonmetric multidimensional scaling. 
The question naturally arises as to how well the solutions coincide. Figures 12.19 and 
12.20 in Example 12.16 respectively give the metric multidimensional scaling 
(principal coordinate analysis) and nonmetric multidimensional scaling solutions 
for the data on universities. The two configurations appear to be quite similar, but a 
quantitative measure would be useful. A numerical comparison of two configura¬ 
tions, obtained by moving one configuration so that it aligns best with the other, is 
called Procrustes analysis , after the innkeeper Procrustes, in Greek mythology, who 
would either stretch or lop off customers’ limbs so they would fit his bed. 
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Constructing the Procrustes Measure of Agreement 

Suppose the n X p matrix X* contains the coordinates of the n points obtained for 
plotting with technique 1 and the n x q matrix Y* contains the coordinates from 
technique 2, where q < p. By adding columns of zeros to Y*, if necessary, we can 
assume that X* and Y* both have the same dimension n x p. To determine how 
compatible the two configurations are, we move, say, the second configuration to 
match the first by shifting each point by the same amount and rotating or reflecting 
the configuration about the coordinate axes. 4 

Mathematically, we translate by a vector b and multiply by an orthogonal 
matrix Q so that the coordinates of the yth point y ■■ are transformed to 

Qy 7 + b 

The vector b and orthogonal matrix Q are then varied to order to minimize the sum, 
over all n points, of squared distances 

d j(*b Qy, + b) = (Xj - Qy, - b)'(x, - Qyj - b) (12-48) 

between x 7 and the transformed coordinates Qy y + b obtained for the second tech¬ 
nique. We take, as a measure of fit, or agreement, between the two configurations, 
the residual sum of squares 

pr2 = q?" % {x > ~ Qy < - b )'(*/- Qy ) ~ b ) (!2^9) 

The next result shows how to evaluate this Procrustes residual sum of squares mea¬ 
sure of agreement and determines the Procrustes rotation of Y* relative to X*. 

Result 12.2 Let the n X p configurations X* and Y* both be centered so that all 
columns have mean zero. Then 

PR 2 = 2 x'jXj + 2 y)y, - 2 £ A, 
y=i j-\ <=i 

= tr[X*X*'] + tr[Y*Y*'] - 2 tr[A] (12-50) 

where A = diag(A!, A 2 , ■.., A p ) and the minimizing transformation is 

A 

Q = X v < u ; = VU' b = 0 (12-51) 

1 = 1 


4 Sibson [30] has proposed a numerical measure of the agreement between two configurations, given 
by the coefficient 

[tr(Y*'X*X*'Y*) 1/2 ] 2 
7 1 “ tr(X*'X*)tr(Y*'Y*) 


For identical configurations, y ~ 0. If necessary, y can be computed after a Procrustes analysis has been 
completed. 
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Here A, U, and V are obtained from the singular-value decomposition 

Y*' X* = U A V' 

ft i (pXn) (nxp) {p*p) (p*p) (p*p) 

Proof. Because the configurations are centered to have zero means ( 2 x / = 0- 
n \ \/~i 

and 2 y/ = 0 1 we have 

/-i / 

X (*/ - Qy ; - ■»)'(*/ - Qy ; - b ) = 2 (*/ “ Qy/)'(*/ - Qy/) + »b'b’ 

/=i 

The last term is nonnegative, so the best fit occurs for b = 0. Consequently, we need 
only consider 

pr 2 = min X (*,■ - Qy/)' (*/• - Qy/) = 2 *;■*/ + 2 y }y/ - 2 “g* 2 X /Qy, 

Q /=1 /=l /-l y /=l 

Using xJQy,- = tr [Qy/Xj], we find that the expression being maximized becomes 

2 */Qy/ = 2 tr tQy y *;] = tr Q 2 y/ x /l 

;=1 /=! L /=! J 

By the singular-value decomposition, 

x y/x; = v*'X* = UAV' = X 

/=! / =1 

where U = [u,, u 2 ,..., u p ] and V = [v t , v 2 ,..., v p ] are p X p orthogonal matrices. 
Consequently, 

X x'Qy ; - tr[o(x A,u,-V( 11 = X A,- tr[Qu,-v-] 

/=i L \i=i / i-i 

The variable quantity in the ith term 

tr[Qu,-v{] = v-Qu, 

has an upper bound of 1 as can be seen by applying the Cauchy-Schwarz inequality 
(2-48) with b = Qv, and d = u ( . That is, since Q is orthogonal, 
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Each of these p terms can be maximized by the same choice Q = VU'. With this 
choice, 


Therefore, 


v-Qu, = vJVU'u/ = [0,..., 0,1,0,..., 0] 1 =1 

0 


-2 max X xJQy/ = -2 (A, + A 2 + • • - + A p ) 


Finally, we verify that QQ' = VU'UV' = VI p V' = I p , so Q is a p X p orthogonal 
matrix, as required. ■ 


Example 12.21 (Procrustes analysis of the data on universities) 7\vo configurations, 
produced by metric and nonmetric multidimensional scaling, of data on universities 
are given Example 12.16.The two configurations appear to be quite close.There is a 
two-dimensional array of coordinates for each of the two scaling methods. Initially, 
the sum of squared distances is 


2 (*/ - y/)' (*/ - y/) = 3.862 

/-i 


A computer calculation gives 


-.9990 .0448 
.0448 ‘ .9990_ 

114.9439 0.000 

0.000 21.3673 


-1.0000 .0076 

.0076 1.0000 


According to Result 12.2, to better align these two solutions, we multiply the non¬ 
metric scaling solution by the orthogonal matrix 


q = 2 v .“: = vu' 

i— i 


.9993 -.0372 
.0372 .9993 


This corresponds to clockwise rotation of the nonmetric solution by about 
2 degrees. After rotation, the sum of squared distances, 3.862, is reduced to the 
Procrustes measure of fit 

25 25 2 

PR 2 = 2 */*/ + 2 yjy/ - 2 X A, = 3.673 ■ 

M /=1 /=! 
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Example 12.22 (Procrustes analysis and additional ordinations of data on forests) 

Data were collected on the populations of eight species of trees growing on ten 
upland sites in southern Wisconsin. These data are shown in Table 12.10. 

The metric, or principal coordinate, solution and nonmetric multidimensional 
scaling solution are shown in Figures 12.26 and 12.27. 


Table 12.10 Wisconsin Forest Data 


Site 

TTee 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

BurOak 

9 

8 

3 

5 

6 

0 

5 

0 

0 

0 

BlackOak 

8 

9 

8 

7 

0 

0 

0 

0 

0 

0 

WhiteOak 

5 

4 

9 

9 

7 

7 

4 

6 

0 

2 

RedOak 

3 

4 

0 

6 

9 

8 

7 

6 

4 

3 

AmericanElm 

2 

2 

4 

5 

6 

0 

5 

0 

2 

5 

Basswood 

0 

0 

0 

0 

2 

7 

6 

6 

7 

6 

Ironwood 

0 

0 

0 

0 

0 

0 

7 

4 

6 

5 

SugarMaple 

0 

0 

0 

0 

0 

5 

4 

8 

8 

9 

Source: See [24]. 



Figure 12.26 Metric multidimensional scaling of the data on forests. 
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4 


2 


0 


-2 


Figure 12.27 Nonmetric multidimensional scaling of the data on forests. 
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This corresponds to clockwise rotation of the nonmetric solution by about 10 degrees. 
After rotation, the sum of squared distances, 8.547, is reduced to the Procrustes 
measure of fit 

10 10 2 

pr 2 = 2 */*/ + 2 fa - 2 2 A - = 6 - 5 " 

/-l ;'-i /-l 

We note that the sampling sites seem to fall along a curve in both pictures. This 
could lead to a one-dimensional nonlinear ordination of the data. A quadratic or 
other curve could be fit to the points. By adding a scale to the curve, we would 
obtain a one-dimensional ordination. 

It is informative to view the Wisconsin forest data when both sampling units and 
variables are shown. A correspondence analysis applied to the data produces the 
plot in Figure 12.28. The biplot is shown in Figure 12.29. 

All of the plots tell similar stories. Sites 1-5 tend to be associated with species of 
oak trees, while sites 7-10 tend to be associated with basswood, ironwood, and sugar 
maples. American elm trees are distributed over most sites, but are more closely 
associated with the lower numbered sites. There is almost a continuum of sites 
distinguished by the different species of trees. ■ 



Figure 12.28 The correspondence analysis plot of the data on forests. 
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Figure 12.29 The biplot of the data on forests. 
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Data Mining 


Introduction 

A very large sample in applications of traditional statistical methodology may mean 
10,000 observations on, perhaps, 50 variables. Today, computer-based repositories 
known as data warehouses may contain many terabytes of data. For some organiza¬ 
tions, corporate data have grown by a factor of 100,000 or more over the last few 
decades. The telecommunications, banking, pharmaceutical, and (package) shipping 
industries provide several examples of companies with huge databases. Consider the 
following illustration. If each of the approximately 17 million books in the Library 
of Congress contained a megabyte of text (roughly 450 pages) in MS Word format, 
then typing this collection of printed material into a computer database would con¬ 
sume about 17 terabytes of disk space. United Parcel Service (UPS) has a package- 
level detail database of about 17 terabytes to track its shipments. 

For our purposes, data mining refers to the process associated with discovering 
patterns and relationships in extremely large data sets. That is, data mining is 
concerned with extracting a few nuggets of knowledge from a relative mountain of 
numerical information. From a business perspective, the nuggets of knowledge rep¬ 
resent actionable information that can be exploited for a competitive advantage. 

Data mining is not possible without appropriate software and fast computers. Not 
suiprisingly, many of the techniques discussed in this book, along with algorithms de¬ 
veloped in the machine learning and artificial intelligence fields, play important roles 
in data mining. Companies with well-known statistical software packages now offer 
comprehensive data mining programs. 5 In addition, special purpose programs such as 
CART have been used successfully in data mining applications. 

Data mining has helped to identify new chemical compounds for prescription 
drugs, detect fraudulent claims and purchases, create and maintain individual 
customer relationships, design better engines and build appropriate inventories, 
create better medical procedures, improve process control, and develop effective 
credit scoring rules. 


5 SAS Institute’s data mining program is currently called Enterprise Miner. SPSS’s data mining 
program is Clementine. 


In traditional statistical applications, sample sizes are relatively small, data are 
carefully collected, sample results provide a basis for inference, anomalies are 
treated but are often not of immediate interest, and models are frequently highly 
structured. In data mining, sample sizes can be huge; data are scattered and histori¬ 
cal (routinely recorded), samples are used for training, validation, and testing (no 
formal inference); anomalies are of interest; and models are often unstructured. 
Moreover, data preparation—including data collection, assessment and cleaning, 
and variable definition and selection—is typically an arduous task and represents 60 
to 80% of the data mining effort. 

Data mining problems can be roughly classified into the following categories: 

• Classification (discrete outcomes): 

Who is likely to move to another cellular phone service? 

• Prediction (continuous outcomes): 

What is the appropriate appraised value for this house? 

• Association/market basket analysis: 

Is skim milk typically purchased with low-fat cottage cheese? 

• Clustering: 

Are there groups with similar buying habits? 

• Description: 

On Thursdays, grocery store consumers often purchase com chips and soft 

drinks together. 

Given the nature of data mining problems, it should not be surprising that many of 
the statistical methods discussed in this book are part of comprehensive data mining 
software packages. Specifically, regression, discrimination and classification proce¬ 
dures (linear rules, logistic regression, decision trees such as those produced by 
CART), and clustering algorithms are important data mining tools. Other tools, 
whose discussion is beyond the scope of this book, include association rules, multi¬ 
variate adaptive regression splines (MARS), ^-nearest neighbor algorithm, neural 
networks, genetic algorithms, and visualization. 6 


The Data Mining Process 

Data mining is a process requiring a sequence of steps. The steps form a strategy 
that is not unlike the strategy associated with any model building effort. Specifically, 
data miners must 

1. Define the problem and identify objectives. 

2. Gather and prepare the appropriate data. 

3. Explore the data for suspected associations, unanticipated characteristics, and 
obvious anomalies to gain understanding. 

4. Clean the data and perform any variable transformation that seems appropriate. 

6 For more information on data mining in general and data mining tools in particular, see the refer¬ 
ences at the end of this chapter. 
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5. Divide the data into training, validation, and, perhaps, test data sets. 

6. Build the model on the training set. 

7. Modify the model (if necessary) based on its performance with the validation data. 

8. Assess the model by checking its performance on validation or test data. 

Compare the model outcomes with the initial objectives. Is the model likely to 

be useful? 

9. Use the model. 

10. Monitor the model performance. Are the results reliable, cost effective? 

In practice, it is typically necessary to repeat one of more of these steps several 
times until a satisfactory solution is achieved. Data mining software suites such as 
Enterprise Miner and Clementine are typically organized so that the user can work 
sequentially through the steps listed and, in fact, can picture them on the screen as a 
process flow diagram. 

Data mining requires a rich collection of tools and algorithms used by a skilled 
analyst with sound subject matter knowledge (or working with someone with sound 
subject matter knowledge) to produce acceptable results. Once established, any suc¬ 
cessful data mining effort is an ongoing exercise. New data must be collected and 
processed, the model must be updated or a new model developed, and, in general, 
adjustments made in light of new experience. The cost of a poor data mining effort 
is high, so careful model construction and evaluation is imperative. 


Model Assessment 

In the model development stage of data mining, several models may be examined 
simultaneously. In the example to follow, we briefly discuss the results of applying 
logistic regression, decision tree methodology, and a neural network to the problem 
of credit scoring (determining good credit risks) using a publicly available data set 
known as the German Credit data. Although the data miner can control the model 
inputs and certain parameters that govern the development of individual models, in 
most data mining applications there is little formal statistical inference. Models are 
ordinarily assessed (and compared) by domain experts using descriptive devices 
such as confusion matrices, summary profit or loss numbers, lift charts, threshold 
charts, and other, mostly graphical, procedures. 

The split of the very large initial data set into training, validation, and testing 
subsets allows potential models to be assessed with data that were not involved in 
model development. Thus, the training set is used to build models that are assessed 
on the validation (holdout) data set. If a model does not perform satisfactorily in the 
validation phase, it is retrained. Iteration between training and validation continues 
until satisfactory performance with validation data is achieved. At this point, a 
trained and validated model is assessed with test data. The test data set is ordinarily 
used once at the end of the modeling process to ensure an unbiased assessment of 
model performance. On occasion, the test data step is omitted and the final assess¬ 
ment is done with the validation sample, or by cross-validation. 

An important assessment tool is the lift chart. Lift charts may be formatted in 
various ways, but all indicate improvement of the selected procedures (models) over 
what can be achieved by a baseline activity. The baseline activity often represents a 


prior conviction or a random assignment. Lift charts are particularly useful for 
comparing the performance of different models. 

Lift is defined as 


Lift 


P (result I condition) 
/’(result) 


If the result is independent of the condition, then Lift = 1. A value of Lift > 1 
implies the condition (generally a model or algorithm) leads to a greater probabili¬ 
ty of the desired result and, hence, the condition is useful and potentially profitable. 
Different conditions can be compared by comparing their lift charts. 


Example 12.23 (A small-scale data mining exercise) A publicly available data set 
known as the German Credit data 7 contains observations on 20 variables for 1000 
past applicants for credit. In addition, the resulting credit rating (“Good” or “Bad”) 
for each applicant was recorded. The objective is to develop a credit scoring rule 
that can be used to determine if a new applicant is a good credit risk or a bad 
credit risk based on values for one or more of the 20 explanatory variables. 
The 20 explanatory variables include CHECKING (checking account status), 
DURATION (duration of credit in months), HISTORY (credit history), AMOUNT 
(credit amount), EMPLOYED (present employment since), RESIDENT (present 
resident since), AGE (age in years), OTHER (other installment debts), INSTALLP 
(installment rate as % of disposable income), and so forth. Essentially, then, we 
must develop a function of several variables that allows us to classify a new appli¬ 
cant into one of two categories: Good or Bad. 

We will develop a classification procedure using three approaches discussed in 
Sections 11.7 and 11.8; logistic regression, classification trees, and neural networks. 
An abbreviated assessment of the three approaches will allow us compare the per¬ 
formance of the three approaches on a validation data set. This data mining exercise 
is implemented using the general data mining process described earlier and SAS 
Enterprise Miner software. 

In the full credit data set, 70% of the applicants were Good credit risks and 30% 
of the applicants were Bad credit risks. The initial data were divided into two sets for 
our purposes, a training set and a validation set. About 60% of the data (581 cases) 
were allocated to the training set and about 40% of the data (419 cases) were allo¬ 
cated to the validation set. The random sampling scheme employed ensured that 
each of the training and validation sets contained about 70% Good applicants and 
about 30% Bad applicants.The applicant credit risk profiles for the data sets follow. 




Credit data 

Training data 

Validation data 


Good: 

700 

401 

299 


Bad: 

300 

180 

120 

** 

Total: 

1000 

581 

419 


7 At the time this supplement was written, the German Credit data were available in a sample data 
Tile accompanying SAS Enterprise Miner. Many other publicly available data sets can be downloaded 
from the following Web site: www.kdnuggets.com. 
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Neural SAMPS 10. 

Network DMAGESCR 


Figure 12.30 The process flow diagram. 


Figure 12.30 shows the process flow diagram from the Enterprise Miner screen. 
The icons in the figure represent various activities in the data mining process. As 
examples, SAMPS 10.DMAGECR contains the data; Data Partition allows the data 
to be split into training, validation, and testing subsets; Transform Variables, as the 
name implies, allows one to make variable transformations; the Regression, Tree, 
and Neural Network icons can each be opened to develop the individual models; 
and Assessment allows an evaluation of each predictive model in terms of predictive 
power, lift, profit or loss, and so on, and a comparison of all models. 

The best model (with the training set parameters) can be used to score a new 
selection of applicants without a credit designation (SAMPSlO.DMAGESCR).The 
results of this scoring can be displayed, in various ways, with Distribution Explorer. 

For this example, the prior probabilities were set proportional to the data; con¬ 
sequently, P(Good) = .7 and P(Bad) = .3. The cost matrix was initially specified 
as follows: 


Actual 


Predicted (Decision) 



Good (Accept) 

Bad (Reject) 

Good 

0 

$i 

Bad 

$5 

0 


so that it is 5 times as costly to classify a Bad applicant as Good (Accept) as it is to 
classify a Good applicant as Bad (Reject). In practice, accepting a Good credit risk 
should result in a profit or, equivalently, a negative cost. To match this formulation 
more closely, we subtract $1 from the entries in the first row of the cost matrix to 
obtain the ‘‘realistic” cost matrix: 


Actual 


Predicted (Decision) 


Good (Accept) 

Bad (Reject) 

Good —$1 

0 

Bad $5 

0 


This matrix yields the same decisions as the original cost matrix, but the results are 
easier to interpret relative to the expected cost objective function. For example, 
after further adjustments, a negative expected cost score may indicate a potential 
profit so the applicant would be a Good credit risk. 

Next, input variables need to be processed (perhaps transformed), models (or 
algorithms) must be specified, and required parameters must be set in all of the icons in 
the process flow diagram.Then the process can be executed up to any point in the dia¬ 
gram by clicking on an icon. All previous connected icons are run. For example, clicking 
on Score executes the process up to and including the Score icon. Results associated 
with individual icons can then be examined by clicking on the appropriate icon. 

We illustrate model assessment using lift charts. These lift charts, available in 
the Assessment icon, result from one execution of the process flow diagram in 
Figure 12.30. 

Consider the logistic regression classifier. Using the logistic regression function 
determined with the training data, an expected cost can be computed for each case 
in the validation set.These expected cost “scores” can then ordered from smallest to 
largest and partitioned into groups by the 10th, 20th,..., and 90th percentiles. The 
first percentile group then contains the 42 (10% of 419) of the applicants with the 
smallest negative expected costs (largest potential profits), the second percentile 
group contains the next 42 applicants (next 10%), and so on. (From a classification 
viewpoint, those applicants with negative expected costs might be classified as Good 
risks and those with nonnegative expected costs as Bad risks.) 

If the model has no predictive power, we would expect, approximately, a uni¬ 
form distribution of, say, Good credit risks over the percentile groups. That is, we 
would expect 10% or .10(299) = 30 Good credit risks among the 42 applicants in 
each of the percentile groups. 

Once the validation data have been scored, we can count the number of Good 
credit risks (of the 42 applicants) actually falling in each percentile group. For 
example, of the 42 applicants in the first percentile group, 40 were actually Good 
risks for a “captured response rate” of 40/299 = .133 or 13.3%. In this case, lift for 
the first percentile group can be calculated as the ratio of the number of Good 
predicted by the model to the number of Good from a random assignment or 


Lift = 


40 

30 


1.33 


The lift value indicates the model assigns 10/299 = .033 or 3.3% more Good risks 
to the first percentile group (largest negative expected cost) than would be assigned 
by chance. 8 

Lift statistics can be displayed as individual (noncumulative) values or as cumu¬ 
lative values. For example, 40 Good risks also occur in the second percentile group 
for the logistic regression classifier, and the cumulative risk for the first two per¬ 
centile groups is 


Lift 


40 + 40 
30 + 30 


1.33 


8 The lift numbers calculated here differ a bit from the numbers displayed in the lift diagrams to fol¬ 
low because of rounding. 
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Figure 12.31 Cumulative lift 
chart for the logistic regression 
classifier. 


The cumulative lift chart for the logistic regression model is displayed in Figure 12.31. 

Lift and cumulative lift statistics can be determined for the classification tree 
tool and for the neural network tool. For each classifier, the entire data set is scored 
(expected costs computed), applicants ordered from smallest score to largest score 
and percentile groups created. At this point, the lift calculations follow those out¬ 
lined for the logistic regression method. The cumulative charts for all three classi¬ 
fiers are shown in Figure 12.32. 


Lift Value 



Percentile ,, J # ^ ; 

i-Tool Name ^ ■ ■■ > , ——j——- 

B Baseline B Neural Q Tree, \ .61 Reg 


Figure 12.32 Cumulative lift 
charts for neural network, 
classification tree, and logistic 
regression tools. 


We see from Figure 12.32 that the neural network and the logistic regression 
have very similar predictive powers and they both do better, in this case, than the 
classification tree. The classification tree, in turn, outperforms a random assignment. 
If this represented the end of the model building and assessment effort, one model 
would be picked (say, the neural network) to score a new set of applicants (without 
a credit risk designation) as Good (accept) or Bad (reject). 

In the decision flow diagram in Figure 12.30, the SAMPS10.DMAGESCR file 
contains 75 new applicants. Expected cost scores for these applicants were created 
using the neural network model. Of the 75 applicants, 33 were classified as Good 
credit risks (with negative expected costs). ■ 

Data mining procedures and software continue to evolve, and it is difficult to 
predict what the future might bring. Database packages with embedded data mining 
capabilities,such as SQL Server 2005, represent one evolutionary direction. 


Exercises 


12.1. Certain characteristics associated with a few recent U.S. presidents are listed in Table 12.11. 


Table 12.11 

President 

Birthplace 
(region of 
United States) 

Elected 

first 

term? 

Party 

Prior U.S. 
congressional 
experience? 

Served as 
vice president? 

1. R. Reagan 

Midwest 

Yes 

Republican 

No 

No 

2. J. Carter 

South 

Yes 

Democrat 

No 

No 

3. G. Ford 

Midwest 

No 

Republican 

Yes 

Yes 

4. R. Nixon 

West 

Yes 

Republican 

Yes 

Yes 

5. L. Johnson 

South 

No 

Democrat 

Yes 

Yes 

6. J. Kennedy 

East 

Yes 

Democrat 

Yes 

No 


(a) Introducing appropriate binary variables, calculate similarity coefficient 1 in 
Table 12.1 for pairs of presidents. 

Hint: You may use birthplace as South, non-South. 

(b) Proceeding as in Part a, calculate similarity coefficients 2 and 3 in Table 12.1 Verify 
the monotonicity relation of coefficients 1,2, and 3 by displaying the order of the 15 
similarities for each coefficient. 

12.2. Repeat Exercise 12.1 using similarity coefficients 5,6, and 7 in Table 12.1. 

12.3. Show that the sample correlation coefficient [see (12-11)] can be written as 

_ _ ad - be _ 

[(a + bUa + c)(£ + d)(c + d)] 1/2 
for two 0-1 binary variables with the following frequencies: 



Variable 2 

0 1 

Variable 1 ^ 

a b 

c d 
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12.4. Show that the monotonicity property holds for the similarity coefficients 1, 2, and 3 in 
Table 12.1. 

Hint: (b + c) — p — (a + d). So, for instance, 

a + d _1_ 

a + d + 2 (b + c) 1 + 2 [p/(a + d) — 1] 

This equation relates coefficients 3 and 1. Find analogous representations for the other 
pairs. 

12.5. Consider the matrix of distances 

12 3 4 


3 11 2 0 

4 L 5 3 4 0 

Cluster the four items using each of the following procedures. 

(a) Single linkage hierarchical procedure. 

(b) Complete linkage hierarchical procedure. 

(c) Average linkage hierarchical procedure. 

Draw the dendrograms and compare the results in (a), (b), and (c). 

12 . 6 . The distances between pairs of five items are as follows: 

1 2 3 4 5 

1 r ° 

2 4 0 

3 6 9 0 

4 1 7 10 0 

5 |_6 3 5 8 0 

Cluster the five items using the single linkage, complete linkage, and average linkage hi¬ 
erarchical methods. Draw the dendrograms and compare the results. 

12.7. Sample correlations for five stocks were given in Example 8.5. These correlations, 
rounded to two decimal places, are reproduced as follows: 



JP Wells Royal Exxon 

Morgan Citibank Fargo DutchShell Mobil 


JP Morgan 

1 




Citibank 

.63 

1 



Wells Fargo 

.51 

.57 

1 


Royal DutchShell 

.12 

.32 

.18 

1 

ExxonMobil 

.16 

.21 

.15 

.68 


12 . 8 . 


Treating the sample correlations as similarity measures, cluster the stocks using the sin¬ 
gle linkage and complete linkage hierarchical procedures. Draw the dendrograms and 
compare the results. 

Using the distances in Example 12.3, cluster the items using the average linkage 
hierarchical procedure. Draw the dendrogram. Compare the results with those in ^ 
Examples 12.3 and 12.5. -5 
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12.9. The vocabulary “richness” of a text can be quantitatively described by counting the 
words used once, the words used twice, and so forth. Based on these counts, a linguist 
proposed the following distances between chapters of the Old Testament book Lamenta¬ 
tions (data courtesy ofY.T.Radday and M. A.Pollatschek): 

Lamentations 

chapter 

1 2 3 4 5 

1 r ° 

Lamentations 2 .76 0 

chapter 3 2.97 .80 0 

4 4.88 4.17 .21 0 

5 l 3.86 1.92 1.51 .51 0 _ 

Cluster the chapters of Lamentations using the three linkage hierarchical methods we 
have discussed. Draw the dendrograms and compare the results. 

12.10. Use Ward’s method to cluster the four items whose measurements on a single variable X 
are given in the following table. 



Measurements 

Item 

X 

1 

2 

2 

1 

3 

5 

4 

8 


(a) Initially, each item is a cluster and we have the clusters 

{1} {2} {3} {4} 

Show that ESS = 0, as it must. 

(b) If we join clusters {1} and {2}, the new cluster {12} has 

ESS, = 2 (*; - *)* = (2 - 1-5) 2 + (1 - 1-5) 2 = -5 

and the ESS associated with the grouping {12}, {3}, {4} is ESS = -5 
+ o + 0 = .5. The increase in ESS (loss of information) from the first step to the 
current step in .5 - 0 = .5. Complete the following table by determining the in¬ 
crease in ESS for all the possibilities at step 2. 



Clusters 

Increase 
in ESS 

{12} 

{3} 

{4} .5 

{13} 

{2} 

{4} 

{14} 

{2} 

{3} 

{1} 

{23} 

{4} 

{1} 

{24} 

{3} 

{1} 

{2} 

{34} 


(c) Complete the last two algamation steps, and construct the dendrogram showing the 
values of ESS at which the mergers take place. 
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12.11. Suppose we measure two variables X l and X 2 for four items A, B, C, and D The data are 
as follows: 



Observations 

Item 


*2 

A 

5 

4 

B 

1 

-2 

C 

-1 

1 

D 

3 

1 


Use the AT-means clustering technique to divide the items into K ~ 2 clusters. Start with 
the initial groups ( AB ) and (CD). 

12.12. Repeat Example 12.11, starting with the initial groups (AC) and ( BD ). Compare your 
solution with the solution in the example. Are they the same? Graph the items in terms 
of their (x\ , x 2 ) coordinates, and comment on the solutions. 

12.13. Repeat Example 12.11, but start at the bottom of the list of items, and proceed up in the 
order D, C, B, A. Begin with the initial groups (AB) and (CD). [The first potential reas¬ 
signment will be based on the distances d 2 (D , (AB)) and d 2 (D y (CD)).] Compare your 
solution with the solution in the example. Are they the same? Should they be the same? 

The following exercises require the use of a computer. 

12.14* Table 11.9 lists measurements on 8 variables for 43 breakfast cereals. 

(a) Using the data in the table, calculate the Euclidean distances between pairs of cereal 
brands. 

(b) Treating the distances calculated in (a) as measures of (dis)similarity, cluster the 
cereals using the single linkage and complete linkage hierarchical procedures. 
Construct dendrograms and compare the results. 

12.15. Input the data inThble 11.9 into a ^f-means clustering program. Cluster the cereals into 
K = 2,3, and 4 groups. Compare the results with those in Exercise 12.14. 

12.16. Tlie national track records data for women are given in Table 1.9. 

(a) Using the data in Thble 1.9, calculate the Euclidean distances between pairs of 
countries. 

(b) Treating the distances in (a) as measures of (dis)similarity, duster the countries using 
the single linkage and complete linkage hierarchical procedures. Construct dendro¬ 
grams and compare the results. 

(c) Input the data in Table 1.9 into a ^-means clustering program. Cluster the countries 
into groups using several values of K. Compare the results with those in Part b. 

12.17. Repeat Exercise 12.16 using the national track records data for men given inThble 8.6. 
Compare the results with those of Exercise 12.16. Explain any differences. 

12.18. Table 12.12 gives the road distances between 12 Wisconsin cities and cities in neighboring 

states. Locate the cities in q = 1,2, and 3 dimensions using multidimensional scaling. Plot 
the minimum stress (q) versus q and interpret the graph. Compare the two-dimensional 
multidimensional scaling configuration with the locations of the cities on a map from an 
atlas. _ 

12.19. Table 12.13 on page 752 gives the “distances” between certain archaeological sites 
from different periods, based upon the frequencies of different types of potsherds found 
at the sites. Given these distances, determine the coordinates of the sites in q = 3,4, 
and 5 dimensions using multidimensional scaling. Plot the minimum stress (q) versus q 
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and interpret the graph. If possible, locate the sites in two dimensions (the first two 
principal components) using the coordinates for the q - 5-dimensional solution. (Treat 
the sites as variables.) Noting the periods associated with the sites, interpret the two- 
dimensional configuration. 

12.20. A sample of n = 1660 people is cross-classified according to mental.health status and 
socioeconomic status in Table 12.14. 

Perform a correspondence analysis of these data. Interpret the results. Can the asso¬ 
ciations in the data be well represented in one dimension? 

12.21. A sample of 901 individuals was cross-classified according to three categories of income 
and four categories of job satisfactiomThe results are given in Table 12.15. 

Perform a correspondence analysis of these data. Interpret the results. 

12.22. Perform a correspondence analysis of the data on forests listed in Table 12.10, and verify 
Figure 12.28 given in Example 12.22. 

12.23. Construct a biplot of the pottery data in Table 12.8. Interpret the biplot. Is the biplot con¬ 
sistent with the correspondence analysis plot in Figure 12.22? Discuss your answer. (Use 
the row proportions as a vector of observations at a site.) 

12.24. Construct a biplot of the mental health and socioeconomic data in Table 12.14. Interpret 
the biplot. Is the biplot consistent with the correspondence analysis plot in Exercise 
12.20? Discuss your answer. (Use the column proportions as the vector of observations 
for each status.) 


Table 12.14 Mental Health Status and Socioeconomic Status Data 


Mental Health Status 

Parental Socioeconomic Status 

A (High) 

B 

C 

D 

E (Low) 

Well 

121 

57 

72 

36 

21 

Mild symptom formation 

188 

105 

141 

97 

71 

Moderate symptom formation 

112 

65 

77 

54 

54 

Impaired 

86 

60 

94 

78 

71 


Source: Adapted from data in Srole, L.,T. S. Langner, S.T. Michael, P. Kirkpatrick,M. K. Opler,and 
T. A. C. Rennie, Mental Health in the Metropolis: The Midtown Manhatten Study , rev. ed. (New York: NYU 
Press, 1978). 


Table 12.15 Income 

and Job Satisfaction Data 

Income 

Job Satisfaction 

Very Somewhat Moderately Very 

dissatisfied dissatisfied satisfied satisfied 

<$25,000 

$25,000-$50,000 

>$50,000 

42 62 184 207 

13 28 81 113 

7 18 54 92 

Source: Adapted from dj 
Wiley, 1990). 

ita inThble 8.2 in Agresti, A., Categorical Data Analysis (New York: John 
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12.25. Using the archaeological data in Table 12.13, determine the two-dimensional metric and 
nonmetric multidimensional scaling plots. (See Exercise 12.19.) Given the coordinates of 
the points in each of these plots, perform a Procrustes analysis. Interpret the results. 

12.26. Table 8.7 contains the Mali family farm data (see Exercise 8.28). Remove the outliers 25, 

34, 69 and 72, leaving at total of n = 72 observations in the data set. Treating the 
Euclidean distances between pairs of farms as a measure of similarity, cluster the farms 
using average linkage and Ward’s method. Construct the dendrograms and compare the 
results. Do there appear to be several distinct clusters of farms? 

12.27. Repeat Exercise 12.26 using standardized observations. Does it make a difference 
whether standardized or unstandardized observations are used? Explain. 

12.28. Using the Mali family farm data inThble 8.7 with the outliers 25,34,69 and 72 removed, 
cluster the farms with the K -means clustering algorithm for K = 5 and K ~ 6. 

Compare the results with those in Exercise 12.26. Is 5 or 6 about the right number of dis¬ 
tinct clusters? Discuss. 

12.29. Repeat Exercise 12.28 using standardized observations. Does it make a difference 
whether standardized of unstandardized observations are used? Explain. 

12.30. A company wants to do a mail marketing campaign. It costs the company $1 for each 
item mailed.They have information on 100,000 customers. Create and interpret a cumu¬ 
lative lift chart from the following information. 

Overall Response Rate: Assume we have no model other than the prediction of the j 

overall response rate which is 20%. That is, if all 100,000 
customers are contacted (at a cost of $100,000), we will re¬ 
ceive around 20,000 positive responses. 

Results of Response Model: A response model predicts who will respond to a 
marketing campaign. We use the response model to as¬ 
sign a score to all 100,000 customers and predict the 
positive responses from contacting only the top 
10,000 customers, the top 20,000 customers, and so 
forth. The model predictions are summarized below. 


Cost 

($) 

Total Customers 
Contacted 

Positive 

Responses 

10000 

10000 

6000 

20000 

20000 

10000 

30000 

30000 

13000 

40000 

40000 

15800 

50000 

50000 

17000 

60000 

60000 

18000 

70000 

70000 

18800 

80000 

80000 

19400 

90000 

90000 

19800 

100000 

100000 

20000 


12.31. Consider the crude-oil data inTable 11.7.Transform the data as in Example 11.14. Ignore 
the known group membership. Using the special purpose software MCLUST , 

(a) select a mixture model using the BIC criterion allowing for the different covariance 
structures listed in Section 12.5 and up to AT = 7 groups. 

(b) compare the clustering results for the best model with the known classifications 
given in Example 11.14. Notice how several dusters correspond to one crude-oil 
classification. 
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Table 1 Standard Normal Probabilities 

Table 2 Student’s r-Distribution Percentage Points 

Table 3 x 2 Distribution Percentage Points 

Table 4 F-Distribution Percentage Points (a = .10) 

Table S F-Distribution Percentage Points (o = .05) 
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.ABLE 1 STANDARD NORMAL PROBABILITIES 



0 z 


z 

.00 

.01 

.02 

.03 

.04 

.05 

.06 

.07 

.08 

.09 

.0 

.5000 

.5040 

.5080 

.5120 

.5160 

.5199 

.5239 

.5279 

.5319 

.5359 

.1 

.5398 

.5438 

.5478 

.5517 

.5557 

.5596 

.5636. 

.5675 

.5714 

.5753 

.2 

.5793 

.5832 

.5871 

.5910 

.5948 

.5987 

.6026 

.6064 

.6103 

.6141 

.3 

.6179 

.6217 

.6255 

.6293 

.6331 

.6368 

.6406 

.6443 

.6480 

.6517 

.4 

.6554 

.6591 

.6628 

.6664 

.6700 

.6736 

.6772 

.6808 

.6844 

.6879 

.5 

.6915 

.6950 

.6985 

.7019 

.7054 

.7088 

.7123 

.7157 

.7190 

.7224 

.6 

.7257 

.7291 

.7324 

.7357 

.7389 

.7422 

.7454 

.7486 

.7517 

.7549 

.7 

.7580 

.7611 

.7642 

.7673 

.7703 

.7734 

.7764 

.7794 

.7823 

.7852 

.8 

.7881 

.7910 

.7939 

.7967 

.7995 

.8023 

.8051 

.8078 

.8106 

.8133 

.9 

,8159 

.8186 

.8212 

.8238 

.8264 

.8289 

.8315 

.8340 

.8365 

.8389 

.0 

.8413 

.8438 

.8461 

.8485 

.8508 

.8531 

.8554 

.8577 

.8599 

.8621 

M 

.8643 

.8665 

.8686 

.8708 

.8729 

.8749 

.8770 

.8790 

.8810 

.8830 

1.2 

.8849 

.8869 

.8888 

.8907 

.8925 

.8944 

.8962 

.8980 

.8997 

.9015 

..3 

.9032 

.9049 

.9066 

.9082 

,9099 

.9115 

.9131 

.9147 

.9162 

.9177 

.4 

.9192 

.9207 

.9222 

.9236 

.9251 

.9265 

.9279 

.9292 

.9306 

.9319 

\5 

.9332 

.9345 

.9357 

.9370 

.9382 

.9394 

.9406 

.9418 

.9429 

.9441 

1.6 

.9452 

.9463 

.9474 

.9484 

.9495 

.9505 

.9515 

.9525 

.9535 

.9545 

-.7 

.9554 

.9564 

.9573 

.9582 

.9591 

.9599 

.9608 

.9616 

.9625 

.9633 

.8 

.9641 

.9649 

.9656 

.9664 

.9671 

.9678 

.9686 

.9693 

.9699 

.9706 

* 9 

.9713 

.9719 

.9726 

.9732 

.9738 

.9744 

.9750 

.9756 

.9761 

.9767 

~.o 

.9772 

.9778 

.9783 

.9788 

.9793 

.9798 

.9803 

.9808 

.9812 

.9817 

1 

.9821 

.9826 

.9830 

.9834 

.9838 

.9842 

.9846 

.9850 

.9854 

.9857 

°2 

.9861 

.9864 

.9868 

.9871 

.9875 

.9878 

.9881 

.9884 

.9887 

.9890 

4.3 

.9893 

.9896 

,9898 

.9901 

.9904 

.9906 

.9909 

.9911 

.9913 

.9916 

~.4 

.9918 

.9920 

.9922 

.9925 

.9927 

.9929 

.9931 

.9932 

.9934 

.9936 

,5 

.9938 

.9940 

.9941 

.9943 

.9945 

.9946 

.9948 

.9949 

.9951 

.9952 


.9953 

.9955 

.9956 

.9957 

.9959 

.9960 

.9961 

.9962 

.9963 

.9964 

2.7 

.9965 

.9966 

.9967 

.9968 

.9969 

.9970 

.9971 

.9972 

.9973 

.9974 

-.8 

.9974 

.9975 

.9976 

.9977 

.9977 

.9978 

.9979 

.9979 

.9980 

.9981 

9 

.9981 

.9982 

.9982 

.9983 

.9984 

.9984 

.9985 

.9985 

.9986 

.9986 

i.O 

.9987 

.9987 

.9987 

.9988 

.9988 

.9989 

.9989 

.9989 

.9990 

.9990 

^.1 

.9990 

.9991 

.9991 

.9991 

,9992 

.9992 

.9992 

.9992 

.9993 

.9993 

.2 

.9993 

.9993 

.9994 

.9994 

.9994 

.9994 

.9994 

.9995 

.9995 

.9995 

"3 

.9995 

.9995 

.9995 

.9996 

.9996 

.9996 

.9996 

.9996 

.9996 

.9997 

i.4 

.9997 

.9997 

.9997 

.9997 

" .9997 

.9997 

.9997 

,9997 

.9997 

.9998 

_.5 

.9998 

.9998 

,9998 

.9998 

.9998 

.9998 

.9998 

.9998 

.9998 

.9998 
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TABLE 2 STUDENT'S t-DISTRIBUTION PERCENTAGE POINTS 







% 

a 









0 ty(a) 

t 




d.£ 





a 





V 

.250 

.100 

.050 

.025 

.010 

.00833 

.00625 

.005 

.0025 

1 

1.000 

3.078 

6.314 

12.706 

31.821 

38.190 

50.923 

63.657 

127321 

2 

.816 

1.886 

2.920 

4.303 

6.965 

7.649 

8.860 

9.925 

14.089 

3 

.765 

1.638 

2.353 

3.182 

4.541 

4.857 

5.392 

5.841 

7.453 

4 

.741 

1.533 

2.132 

2.776 

3.747 

3.961 

4.315 

4.604 

5.598 

5 

.727 

1.476 

2.015 

2.571 

3.365 

3.534 

3.810 

4.032 

4.773 

6 

.718 

1.440 

1.943 

2.447 

3.143 

3.287 

3.521 

3.707 

4317 

7 

.711 

1.415 

1.895 

2.365 

2.998 

3.128 

3.335 

3.499 

4.029 

8 

.706 

1.397 

1.860 

2.306 

2.896 

3.016 

3.206 

3.355 

3.833 

9 

.703 

1.383 

1.833 

2.262 

2.821 

2.933 

3.111 

3.250 

3.690 

10 

.700 

1.372 

1.812 

2.228 

2.764 

2.870 

3.038 

3.169 

3.581 

11 

.697 

1.363 

1.796 

2.201 

2.718 

2.820 

2.981 

3.106 

3.497 

12 

.695 

1.356 

1.782 

2.179 

2.681 

2.779 

2,934 

3.055 

3.428 

13 

.694 

1.350 

1.771 

2.160 

2.650 

2.746 

2.896 

3.012 

3.372 

14 

.692 

1.345 

1.761 

2.145 

2.624 

2.718 

2.864 

2.977 

3.326 

15 

.691 

1.341 

1.753 

2.131 

2.602 

2.694 

2.837 

2.947 

3.286 

16 

.690 

1.337 

1.746 

2.120 

2.583 

2.673 

2.813 

2.921 

3.252 

17 

.689 

1.333 

1.740 

2.110 

2.567 

2.655 

2.793 

2.898 

3.222 

18 

.688 

1.330 

1.734 

2.101 

2.552 

2.639 

2.775 

2.878 

3.197 

19 

.688 

1.328 

1.729 

2.093 

2.539 

2.625 

2.759 

2.861 

3.174 

20 

.687 

1.325 

1.725 

2.086 

2.528 

2.613 

2.744 

2.845 

3.153 

21 

.686 

1.323 

1.721 

2.080 

2.518 

2.601 

2.732 

2.831 

3.135 

22 

.686 

1.321 

1.717 

2.074 

2.508 

2.591 

2.720 

2.819 

3.119 

23 

.685 

1.319 

1.714 

2.069 

2.500 

2.582 

2.710 

2.807 

3.104 

24 

.685 

1.318 

1.711 

2.064 

2.492 

2.574 

2.700 

2.797 

3.091 

25 

.684 

1.316 

1.708 

2.060 

2.485 

2.566 

2.692 

2.787 

3.078 

26 

.684 

1.315 

1.706 

2.056 

2.479 

2.559 

2.684 

2.779 

3.067 

27 

.684 

1.314 

1.703 

2.052 

2.473 

2.552 

2.676 

2.771 

3.057 

28 

.683 

1.313 

1.701 

2.048 

2.467 

2.546 

2.669 

2.763 

3.047 

29 

.683 

1.311 

1.699 

2.045 

2.462 

2.541 

2.663 

2.756 

3.038 

30 

.683 

1.310 

1.697 

2.042 

2.457 

2.536 

2.657 

2.750 

3.030 

40 

.681 

1.303 

1.684 

2.021 

2.423 

2.499 

2.616 

2.704 

2.971 

60 

.679 

1:296 

1.671 

2.000 

2.390 

2.463 

2.575 

2.660 

2.915 

120 

.677 

1.289 

1.658 

1.980 

2.358 

2.428 

2,536 

2.617 

2.860 

00 

.674 

1.282 

1.645 

1.960 

2.326 

2.394 

2.498 

2.576 

2.813 
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.990 

.950 

.900 

.500 

.0002 

.004 

.02 

.45 

.02 

.10 

.21 

1.39 

.11 

.35 

.58 

2.37 

.30 

.71 

1.06 

3.36 

.55 

1.15 

1.61 

4.35 

.87 

1.64 

2.20 

5.35 

1.24 

2.17 

2.83 

6.35 

1.65 

2.73 

3.49 

7.34 

2.09 

3.33 

4.17 

8.34 

2.56 

3.94 

4.87 

9.34 


3.05 
" 3.57 

13 4.11 

4.66 
5.23 
^ 5.81 

1 / 6.41 

7.01 
7.63 
8.26 
zi 8.90 

9.54 
10.20 
24 10.86 

^ 11.52 

12.20 
12.88 
7X 13.56 

14.26 
14.95 
22.16 
xi 29.71 

37.48 
45.44 
53.54 
u 61.75 

70.06 


2.71 

4.61 

6.25 

7.78 

9.24 

10.64 

12.02 

13.36 

14.68 

15.99 

17.28 

18.55 

19.81 
21.06 

22.31 
23.54 

24.77 

25.99 

27.20 

28.41 
29.62 

30.81 
32.01 

33.20 
34.38 

35.56 

36.74 

37.92 
39.09 
40.26 

51.81 

63.17 
74.40 

85.53 

96.58 

107.57 

118.50 


3.84 

5.99 

7.81 

9.49 

11.07 

12.59 
14.07 
15.51 

16.92 

18.31 

19.68 
21.03 

22.36 

23.68 
25.00 
26.30 

27.59 
28.87 

30.14 

31.41 

32.67 

33.92 

35.17 

36.42 

37.65 
38.89 

40.11 

41.34 

42.56 

43.77 
55.76 
67.50 
79.08 

90.53 

101.88 

113.15 

124.34 


5.02 

7.38 

9.35 

11.14 

12.83 

14.45 
16.01 

17.53 
19.02 

20.48 

21.92 

23.34 

24.74 

26.12 

27.49 

28.85 

30.19 

31.53 

32.85 

34.17 
35.48 

36.78 
38.08 

39.36 

40.65 

41.92 

43.19 

44.46 

45.72 
46.98 

59.34 

71.42 
83.30 
95.02 

106.63 

118.14 

129.56 


6.63 

9.21 

11.34 

13.28 
15.09 

16.81 
18.48 
20.09 

21.67 

23.21 

24.72 

26.22 

27.69 

29.14 

30.58 
32.00 
33.41 

34.81 

36.19 

37.57 

38.93 

40.29 

41.64 
42.98 

44.31 

45.64 
46.96 

48.28 

49.59 
50.89 

63.69 

76.15 
88.38 

100.43 

112.33 

124.12 

135.81 


7.88 

10.60 

12.84 

14.86 

16.75 

18.55 

20.28 

21.95 
23.59 

25.19 

26.76 

28.30 
29.82 

31.32 

32.80 
34.27 

35.72 

37.16 

38.58 
40.00 
41.40 

42.80 

44.18 

45.56 

46.93 

48.29 

49.64 

50.99 

52.34 

53.67 

66.77 

79.49 

91.95 
104.21 

116.32 

128.30 
140.17 
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TABLE 4 F-DISTRIBUTION PERCENTAGE POINTS (a = .10) 



39.86 

49.50 

5339 

55.83 

57.24 

58.20 

58.91 

59.44 

59.86 

60.19 

60.71 

6132 

61.74 

62.05 

62.26 62.53 

62.79 

8.53 

9.00 

9.16 

9.24 

9.29 

9.33 

9.35 

9.37 

9.38 

9.39 

9.41 

9.42 

9.44 

9.45 

9.46 

9.47 

9.47 

5.54 

5.46 

5.39 

534 

5.31 

5.28* 

5.27 

5.25 

5.24 

5.23 

5.22 

5.20 

5.18 

5.17 

5.17 

5.16 

5.15 

4.54 

4.32 

4.19 

4.11 

4.05 

4.01 

3.98 

3.95 

3.94 

3.92 

3.90 

3.87 

3.84 

3.83 

3.82 

3.80 

3.79 

4.06 

3.78 

3.62 

3.52 

3.45 

3.40 

3.37 

3.34 

3.32 

3.30 

337 

334 

3.21 

3.19 

3.17 

3.16 

3.14 

3.78 

3.46 

3.29 

3.18 

3.11 

3.05 

3.01 

2.98 

2.96 

2.94 

2.90 

2.87 

2.84 

2.81 

2.80 

2.78 

2.76 

3.59 

3.26 

3.07 

2.96 

2.88 

2.83 

2.78 

2.75 

2.72 

2.70 

2.67 

2.63 

2.59 

2.57 

236 

2.54 

2.51 

3.46 

3.11 

2.92 

2.81 

2.73 

2.67 

2.62 

2.59 

2.56 

2.54 

2.50 

2.46 

2.42 

2.40 

238 

2,36 

2.34 

336 

3.01 

2.81 

2.69 

2.61 

2.55 

2.51 

2.47 

2.44 

2.42 

2.38 

2.34 

230 

2.27 

2.25 

2.23 

2.21 

3.29 

2.92 

2.73 

2.61 

2.52 

2.46 

2.41 

2.38 

2.35 

2.32 

2.28 

234 

2.20 

2.17 

2.16 

2.13 

2.11 

3.23 

2.86 

2.66 

2.54 

2.45 

2.39 

2.34 

2.30 

2.27 

2.25 

2.21 

2.17 

2.12 

2.10 

2.08 

2.05 

2.03 

3.18 

2.81 

2.61 

2.48 

2.39 

2.33 

2.28 

234 

2.21 

2.19 

2.15 

2.10 

2.06 

2.03 

2.01 

1.99 

1.96 

3.14 

2.76 

2.56 

2.43 

' 2.35 

2.28 

2.23 

230 

2.16 

2.14 

2.10 

2.05 

2.01 

1.98 

1.96 

1.93 

1.90 

3.10 

2.73 

2.52 

2.39 

2.31 

2.24 

2.19 

2.15 

2.12 

2.10 

2.05 

2.01 

1.96 

1.93 

1.91 

1.89 

1.86 

3.07 

2.70 

2.49 

236 

2.27 

231 

2.16 

2.12 

2.09 

2.06 

2.02 

1.97 

1.92 

1.89 

1.87 

1.85 

1.82 

3.05 

2.67 

2.46 

2.33 

2.24 

2.18 

2.13 

2.09 

2.06 

2.03 

1.99 

1.94 

1.89 

1.86 

1.84 

1.81 

1.78 

3.03 

2.64 

2.44 

2.31 

2.22 

2.15 

2.10 

2.06 

2.03 

2.00 

1.96 

1.91 

1.86 

1.83 

1.81 

1.78 

1.75 

3.01 

2.62 

2.42 

2.29 

2.20 

2.13 

2.08 

2.04 

2.00 

1.98 

1.93 

1.89 

1.84 

1.80 

1.78 

1.75 

1.72 

2.99 

2.61 

2.40 

2.27 

2.18 

2.11 

2.06 

2.02 

1.98 

1.96 

1.91 

1.86 

1.81 

1.78 

1.76 

1.73 

1.70 

2.97 

2.59 

2.38 

2.25 

2.16 

2.09 

2.04 

2.00 

1.96 

1.94 

1.89 

1.84 

1.79 

1.76 

1.74 

1.71 

1.68 

2.96 

2.57 

2.36 

2.23 

2.14 

2.08 

2.02 

1.98 

1.95 

1.92 

1.87 

1.83 

1.78 

1.74 

1.72 

1.69 

1.66 

2.95 

2.56 

2.35 

2.22 

2.13 

2.06 

2.01 

1.97 

1.93 

1.90 

1.86 

1.81 

1.76 

1.73 

1.70 

1.67 

1.64 

2,94 

2.55 

234 

231 

2.11 

2.05 

1.99 

1.95 

1.92 

1.89 

1.84 

1.80 

1.74 

1.71 

1.69 

1.66 

1.62 

2.93 

2.54 

2.33 

2.19 

2.10 

2.04 

1.98 

1.94 

1.91 

1.88 

1.83 

1.78 

1.73 

1.70 

1.67 

1.64 

1.61 

2.92 

253 

2.32 

2.18 

2.09 

2.02 

1.97 

1.93 

1.89 

1.87 

1.82 

1.77 

1.72 

1.68 

1.66 

1.63 

1.59 

2.91 

2.52 

2.31 

2.17 

2.08 

2.01 

1.96 

1.92 

1.88 

1.86 

1.81 

1.76 

1.71 

1.67 

1.65 

1.61 

1.58 

2.90 

2.51 

2.30 

2.17 

2.07 

2.00 

1.95 

1.91 

1.87 

1.85 

1.80 

1.75 

1.70 

1.66 

1.64 

1.60 

1.57 

2.89 

2.50 

2.29 

2.16 

2.06 

2.00 

1.94 

1.90 

1.87 

1.84 

1.79 

1.74 

1.69 

1.65 

1.63 

1.59 

1.56 

2.89 

2.50 

2.28 

2.15 

2.06 

1.99 

1.93 

1.89 

1.86 

1.83 

1.78 

1.73 

1.68 

1.64 

1.62 

1.58 

1.55 

2.88 

2.49 

2.28 

2.14 

2.05 

1.98 

1.93 

1.88 

1.85 

1.82 

1.77 

1.72 

1.67 

1.63 

1.61 

1.57 

1.54 

2.84 

2.44 

2.23 

2.09 

2.00 

1.93 

1.87 

1.83 

1.79 

1.76 

1.71 

1.66 

1.61 

1.57 

1.54 

1.51 

1.47 

2.79 

239 

2.18 

2.04 

1.95 

1.87 

1.82 

1.77 

1.74 

1.71 

1.66 

1.60 

1.54 

1.50 

1.48 

1.44 

1.40 

2,75 

235 

2.13 

1.99 

1.90 

1.82 

1.77 

1.72 

1.68 

1.65 

1.60 

1.55 

1.48 

1.45 

1.41 

137 

1.32 

2.71 

2.30 

2.08 

1.94 

1.85 

1.77 

1.72 

1.67 

1.63 

1.60 

1.55 

1.49 

1.42 

1.38 

1.34 

1.30 

1.24 
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■ABLE 5 f-DISTRIBUTION PERCENTAGE POINTS (a = .05) 



\ 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

12 

15 * 

20 

25 


40 

60 

1 

161.5 199.5 215.7 224.6 230.2 234.0 236.8 238.9 240.5 241.9 243.9 246.0 248.0 249.3 250.1 251.1 252.2 \ 


18.51 

19.00 

19.16 

19.25 19.30 

19.33 

19.35 

19.37 

19.38 

19.40 

19.41 

19.43 

19.45 

19.46 

19.46 19.47 

19.48 


10.13 

9.55 

9.28 

9.12 

9.01 

8.94 

8.89 

8.85 

8.81 

8.79 

8.74 

8.70 

8.66 

8.63 

8.62 

8.59 

8.57 

4 

7.71 

6.94 

6.59 

6.39 

6.26 

6.16 

6.09 

6.04 

6.00 

5.96 

5.91 

5.86 

5.80 

5.77 

5.75 

5.72 

5.69 

j 

6.61 

5.79 

5.41 

5.19 

5.05 

4.95 

4.88 

4.82 

4.77 

4.74 

4.68 

4.62 

4.56 

4.52 

4.50 

4.46 

4.43 

t 

5.99 

5.14 

4.76 

4.53 

4-39 

4.28 

4.21 

4.15 

4.10 

4.06 

4.00 

3.94 

3.87 

3.83 

3.81 

3.77 

3.74 

1 

5.59 

4.74 

4.35 

4.12 

3.97 

3.87 

3.79 

3.73 

3.68 

3.64 

3.57 

3.51 

3.44 

3.40 

3.38 

3.34 

3.30 

8 

532 

4.46 

4.07 

3.84 

3.69 

3.58 

3.50 

3.44 

3.39 

3.35 

3.28 

3.22 

3.15 

3.11 

3.08 

3.04 

3.01 

9 

5.12 

4.26 

3.86 

3.63 

3.48 

3.37 

3.29 

3.23 

3.18 

3.14 

3.07 

3.01 

2.94 

2.89 

2.86 

2.83 

2.79 

) 

4.96 

4.10 

3.71 

3.48 

3.33 

3.22 

3.14 

3.07 

3.02 

2.98 

2.91 

2.85 

2.77 

2.73 

2.70 

2.66 

2.62 


4.84 

3.98 

3.59 

3.36 

3.20 

3.09 

3.01 

2.95 

2.90 

2.85 

2.79 

2.72 

2.65 

2.60 

2-57 

2.53 

2.49 

12 

4.75 

3.89 

3.49 

3.26 

3.11 

3.00 

2.91 

2.85 

2.80 

2.75 

2.69 

2.62 

2.54 

2.50 

2.47 

2.43 

238 


4.67 

3.81 

3.41 

3.18 

3.03 

2.92 

2.83 

2.77 

2.71 

2.67 

2.60 

2.53 

2.46 

2.41 

2.38 

2.34 

2.30 

, 

4.60 

3.74 

3.34 

3.11 

2.96 

2.85 

2.76 

2.70 

2.65 

2.60 

2.53 

2.46 

2.39 

2.34 

2.31 

2.27 

2.22 

5 

4.54 

3.68 

3.29 

3.06 

2.90 

2.79 

2.71 

2.64 

2.59 

2.54 

2.48 

2.40 

2.33 

2.28 

2.25 

2.20 

2.16 

16 

4.49 

3.63 

3.24 

3.01 

2.85 

2.74 

2.66 

2.59 

2.54 

2.49 

2.42 

2.35 

2.28 

2.23 

2.19 

2.15 

2.11 

J 

4.45 

3.59 

3.20 

2.96 

2.81 

2.70 

2.61 

2.55 

2.49 

2.45 

2.38 

2.31 

2.23 

2.18 

2.15 

2.10 

2.06 

; 

4.41 

3.55 

3.16 

2.93 

2.77 

2.66 

2.58 

2.51 

2.46 

2.41 

2.34 

2.27 

2.19 

2.14 

2.11 

2.06 

2.02 

■ i 

4.38 

3.52 

3.13 

2.90 

2.74 

2.63 

2.54 

2.48 

2.42 

2.38 

2.31 

2.23 

2.16 

2.11 

2.07 

2.03 

1.98 


4.35 

3.49 

3.10 

2.87 

2.71 

2.60 

2.51 

2.45 

2.39 

2.35 

2.28 

2.20 

2.12 

2.07 

2.04 

1.99 

1.95 


4.32 

3.47 

3.07 

2.84 

2.68 

2.57 

2.49 

2.42 

2.37 

2.32 

2.25 

2.18 

2.10 

2.05 

2.01 

1.96 

1.92 

1 

4.30 

3.44 

3.05 

2.82 

2.66 

2.55 

2.46 

2.40 

2.34 

2.30 

2.23 

2.15 

2.07 

2.02 

1.98 

1.94 

i .89 


4.28 

3.42 

3.03 

2.80 

2.64 

2.53 

2.44 

2.37 

2.32 

2.27 

2.20 

2.13 

2.05 

2.00 

1.96 

1.91 

1.86 

24 

4.26 

3.40 

3.01 

2.78 

2.62 

2.51 

2.42 

2.36 

2.30 

2.25 

2.18 

2.11 

2.03 

1.97 

1.94 

1.89 

1.84 

*.5 

4.24 

3.39 

2.99 

2.76 

2.60 

2.49 

2.40 

2.34 

2.28 

2.24 

2.16 

2.09 

2.01 

1.96 

1.92 

1.87 

1.82 ; 

j 

4.23 

3.37 

2.98 

2.74 

2.59 

2.47 

2.39 

2.32 

2.27 

2.22 

2.15 

2.07 

1.99 

1.94 

1.90 

1.85 

1.80 

*7 

4.21 

3.35 

2.96 

2.73 

2.57 

2.46 

2.37 

2.31 

2.25 

2.20 

2.13 

2.06 

1.97 

1.92 

1.88 

1.84 

1.79 i 

78 

4.20 

3.34 

2.95 

2.71 

2.56 

2.45 

2.36 

2.29 

2.24 

2.19 

2.12 

2.04 

1.96 

1.91 

1.87 

1.82 

1.77 | 


4.18 

3.33 

2.93 

2.70 

2.55 

2.43 

2.35 

2.28 

2.22 

2.18 

2.10 

2.03 

1.94 

1.89 

1.85 

1.81 

1.75 S 

J 

) 

4.17 

3.32 

2.92 

2.69 

2.53 

2.42 

2.33 

2.27 

2.21 

2.16 

2.09 

2.01 

1.93 

1.88 

1.84 

1.79 

1.74 

0 

4.08 

3.23 

2.84 

2.61 

2.45 

2.34 

2.25 

2.18 

2.12 

2.08 

2.00 

1.92 

1.84 

1.78 

1.74 

1.69 

1.64 


4.00 

3.15 

2.76 

2.53 

2.37 

2.25 

2.17 

2.10 

2.04 

1.99 

1.92 

1.84 

1.75 

1.69 

1.65 

1.59 

1.53 


3.92 

3.07 

2.68 

2.45 

2.29 

2.18 

2.09 

2.02 

1.96 

1.91 

1.83 

1.75 

1.66 

1.60 

1.55 

1.50 

1.43 

> 

3.84 

3.00 

2.61 

2.37 

2.21 

2.10 

2.01 

1.94 

1.88 

1.83 

1.75 

1.67 

1.57 

1.51 

1.46 

1.39 

1.32 


TABLE 6 f-DISTRIBUTION PERCENTAGE POINTS (a = .01) 




1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

12 

15 

20 

25 

30 

40 

60 

1 

4052 . 5000 . 5403 . 5625 . 5764 . 5859 . 5928 . 5981 . 6023 . 6056 . 6106 . 6157 . 6209 . 6240 . 6261 . 6287 . 6313 . 

2 

98.50 

99.00 

99.17 

99.25 

99.30 

99.33 

99.36 

99.37 

99.39 

99.40 

99.42 

99.43 

99.45 

99.46 

99.47 

99.47 

99.48 

3 

34.12 

30.82 

29.46 

28.71 

28.24 

27.91 

27.67 

27.49 

27.35 

27.23 

27.05 

26.87 

26.69 

26.58 

26.50 

26.41 

26.32 

4 

21.20 

18.00 

16.69 

15.98 

15.52 

15.21 

14.98 

14.80 

14.66 

1455 

14.37 

14.20 

14.02 

13.91 

13.84 

13.75 

13.65 

5 

16.26 

13.27 

12.06 

11.39 

10.97 

10.67 

10.46 

10.29 

10.16 

10.05 

9.89 

9.72 

9.55 

9.45 

9.38 

9.29 

9.20 

i 6 

13.75 

10.92 

9.78 

9.15 

8.75 

8.47 

8.26 

8.10 

7.98 

7.87 

7.72 

7.56 

7.40 

7.30 

7.23 

7.14 

7.06 

! 7 

12.25 

9.55 

8.45 

7.85 

7.46 

7.19 

6.99 

6.84 

6.72 

6.62 

6.47 

6.31 

6.16 

6.06 

5.99 

5.91 

5.82 

! 8 

11.26 

8.65 

7.59 

7.01 

6.63 

6.37 

6.18 

6.03 

5.91 

5.81 

5.67 

5.52 

5.36 

5.26 

5.20 

5.12 

5.03 

j 9 

10.56 

8.02 

6.99 

6.42 

6.06 

5.80 

5.61 

5.47 

5.35 

5.26 

5.11 

4.96 

4.81 

4.71 

4.65 

4-57 

4-48 

10 

10.04 

7.56 

6.55 

5.99 

5.64 

5.39 

5.20 

5.06 

4.94 

4.85 

4.71 

456 

4.41 

4.31 

4.25 

4.17 

4.08 

! 11 

9.65 

7.21 

6.22 

5.67 

5.32 

5.07 

4.89 

4.74 

4.63 

454 

4.40 

4.25 

4.10 

4.01 

3.94 

3.86 

3.78 

12 

9.33 

6.93 

5.95 

5.41 

5.06 

4.82 

4.64 

4.50 

4.39 

4.30 

4.16 

4.01 

3.86 

3.76 

3.70 

3.62 

3.54 

13 

9.07 

6.70 

5.74 

5.21 

4.86 

4.62 

4.44 

4.30 

4.19 

4.10 

3.96 

3.82 

3.66 

357 

3.51 

3.43 

3.34 

14 

8.86 

6.51 

5.56 

5.04 

4.69 

4.46 

4.28 

4.14 

4.03 

3.94 

3.80 

3.66 

3.51 

3.41 

3.35 

3.27 

3-18 

15 

8.68 

6.36 

5.42 

4.89 

4.56 

4.32 

4.14 

4.00 

3.89 

3.80 

3.67 

3.52 

3.37 

3.28 

3.21 

3.13 

3.05 

16 

8.53 

6.23 

5.29 

4.77 

4.44 

420 

4.03 

3.89 

3.78 

3.69 

3.55 

3.41 

3.26 

3.16 

3.10 

3.02 

2.93 

17 

8.40 

6.11 

5.19 

4.67 

4.34 

4.10 

3.93 

3.79 

3.68 

3.59 

3.46 

3.31 

3.16 

3.07 

3.00 

2.92 

2.83 

1 18 

8.29 

6.01 

5.09 

4.58 

4.25 

4.01 

3.84 

3.71 

3.60 

3.51 

3.37 

3.23 

3.08 

2.98 

2.92 

2.84 

2.75 

19 

8.18 

5.93 

5.01 

4.50 

4.17 

3.94 

3.77 

3.63 

3.52 

3.43 

3.30 

3.15 

3.00 

2.91 

2.84 

2.76 

2.67 

20 

8.10 

5.85 

4.94 

4.43 

4.10 

3.87 

3.70 

3.56 

3.46 

3.37 

3.23 

3.09 

2.94 

2.84 

2.78 

2.69 

2.61 

21 

8.02 

5.78 

4.87 

4.37 

4.04 

3.81 

3.64 

3.51 

3.40 

3.31 

3.17 

3.03 

2.88 

2.79 

2.72 

2.64 

2.55 

22 

7.95 

5.72 

4.82 

4.31 

3.99 

3.76 

3.59 

3.45 

3.35 

3.26 

3.12 

2.98 

2.83 

2.73 

2.67 

2.58 

2.50 

23 

7.88 

5.66 

4.76 

4.26 

3.94 

3.71 

3.54 

3.41 

3.30 

3.21 

3.07 

2.93 

2.78 

2.69 

2.62 

2.54 

2.45 

24 

7.82 

5.61 

4.72 

422 

3.90 

3.67 

3.50 

3.36 

3.26 

3.17 

3.03 

2.89 

2.74 

2.64 

2.58 

2.49 

2.40 

25 

7.77 

5.57 

4.68 

4.18 

3.85 

3.63 

3.46 

3.32 

3.22 

3.13 

2.99 

2.85 

2.70 

2.60 

2.54 

2.45 

2.36 

26 

7.72 

5.53 

4.64 

4.14 

3.82 

3.59 

3.42 

3.29 

3.18 

3.09 

2.96 

2.81 

2.66 

2.57 

2.50 

2.42 

2.33 

27 

7.68 

5.49 

4.60 

4.11 

3.78 

3.56 

3.39 

3.26 

3.15 

3.06 

2.93 

2.78 

2.63 

2.54 

2.47 

2.38 

2.29 

28 

7.64 

5.45 

4.57 

4.07 

3.75 

3.53 

3.36 

3.23 

3.12 

3.03 

2.90 

2.75 

2.60 

2.51 

2.44 

2.35 

2.26 

29 

7.60 

5.42 

4.54 

4.04 

3.73 

3.50 

3.33 

3.20 

3.09 

3.00 

2.87 

. 2.73 

2.57 

2.48 

2.41 

2.33 

2.23 

30 

7.56 

5.39 

4.51 

4.02 

3.70 

3.47 

3.30 

3.17 

3.07 

2.98 

2.84 

2.70 

2.55 

2.45 

2.39 

2.30 

2.21 

40 

7.31 

5.18 

4.31 

3.83 

3.51 

3.29 

3.12 

2.99 

2.89 

2.80 

2.66 

2.52 

2.37 

2.27 

2.20 

2.11 

2.02 

60 

7.08 

4.98 

4.13 

3.65 

3.34 

3.12 

2.95 

2.82 

2.72 

2.63 

2.50 

2.35 

2.20 

2.10 

‘ 2.03 

1794 

1.84 

120 

6.85 

4.79 

3.95 

3.48 

3.17 

2.96 

2.79 

2.66 

256 

2.47 

2.34 

2.19 

2.03 

1.93 

1.86 

1.76 

1.66 

oo 

6.63 

4.61 

3.78 

3.32 

3.02 

2.80 

2.64 

251 

2.41 

2.32 

2.18 

2.04 

1.88 

1.78 

1.70 

1.59 

1.47 





1 


Data Index 


765 


Data Index 


Admission, 661 
examples, 614, 660 
Airline distances 710 
example, 709 
Air pollution, 39 
examples, 39, 206, 425,474, 535 
Amitriptyline, 426 
example, 426 
Anaconda snake, 357 
example, 356 

Archeological site distances 752 

examples, 750,754 

Bankruptcy, 657 

examples, 45,656,658 
Battery failure, 424 
example, 424 
Biting fly, 352 
example, 350 
Bonds 346 
example, 345 

Bones (mineral content), 43,353 

examples, 41,207,268, 350, 351, 

425,476 

Breakfast cereal, 666 
examples, 45, 665, 750 

Bull, 46 

examples 46, 207,425, 476, 537, 665 

Calcium (bones), 329, 330 
example, 331 

Carapace (painted turtles), 344, 532 
examples 343,356,445,454,532 
Car body assembly, 271 

examples 270,480 


Census tract, 474 
examples 443,474,535 
College test scores 228 
examples, 226, 267,423 
Computer requirements, 380,400 
examples 380, 383,400,405,408, 
410, 412 

Concho water snake, 668 

example, 665 
Crime, 569 
example, 569 
Crude oil, 662 
examples 347, 356,625, 

661,754 

Diabetic, 572 
example, 572 

Effluent, 276 
examples 276, 337, 338 
Egyptian skull, 349 
examples, 269, 347 
Electrical consumption, 289 
examples, 289, 293,295, 338, 356 
Electrical time-of-use pricing, 350 
example, 349 
Energy consumption, 147 
examples 147, 270 
Examination scores 505 
example, 505 

Female bear, 24 

examples 24,262 
Forest, 736 
examples 736,753 


Fowl, 521 

examples 520, 532,552,559 

Grizzly bear, 262,478 
examples 262, 478 

Hair (Peruvian), 263 

example, 263 
Hemophilia, 587, 664,665 
examples, 587, 591, 663 
Hook-billed kite, 268, 346 
examples, 268, 344 

Iris, 658 

examples 347,619,645,658,660, 705 

Job satisfaction/characteristics 

555,753 

examples, 553,563, 565, 753 

Lamentations 749 
example, 749 
Largest companies 38 
examples 38,183,205, 206, 423,471 
Lizard*—two genera, 335 
example, 334 
Lizard size, 17 
examples, 17,18 
Love and marriage, 326 
example, 325 
Lumber, 267 
example, 267 

Mali family farm, 479 
examples, 479, 538, 754 
Mental health, 753 
example, 753 
Mice, 453,475 
examples, 453, 458,475, 537 
Milk transportation cost, 269, 345 
examples 45,268, 343 
Multiple sclerosis 42 
examples, 41, 207,656 
Musical aptitude, 236 
example, 236 

National parks 47 

examples 46, 208 


National track records 44, 477 

examples, 43, 207, 357, 476, 

537, 750 

Natural gas 414 
example, 413 
Number parity, 342 
example, 342 
Numerals 679 
examples 678,684,687, 690 
Nursing home, 306-07 
examples, 306, 309,311 

Olympic decathlon, 499 

examples 499,511,573 
Overtime (police), 240, 478 
examples, 239,242,244, 248, 269, 
270, 460, 463,464,478 
Oxygen consumption, 348 
examples, 45, 347 

Paper quality, 15 

examples, 14,20, 207 

Peanut, 354 

example, 353 
Plastic film, 318 
example, 318 
Pottery, 716 
examples 716, 753 
Profitability, 533 
examples 533,571 
Psychological profile, 207 
examples 207, 478,537 
Public utility, 688 
examples 26,28, 45,46,688,690, 
699, 711, 726 

Pulp and paper properties 427 

examples, 427, 478,537, 538, 573 

Radiation, 180,198 
examples 180,197,206,221,226, 
233,261 

Radiotherapy, 42 
examples, 41, 207,475 
Reading/arithmetic test scores 

569 

example, 569 
Real estate, 372 
examples, 372,423 




766 


Data Index 


Relay tower breakdowns, 358,428 
examples, 357, 427 
Road distances, 751 

example, 750 

Salmon, 604 

examples, 603, 639, 663, 669 
Sleeping dog, 282 
example, 281 
Smoking, 573 
example, 572 
Snow removal, 148 
examples, 148,208,270 
Spectral reflectance, 355 
examples, 354, 355 
Spouse, 351 
example, 350 



Stiffness (lumber), 186,190 

i 


examples, 186,190, 342, 



535,571 



Stock price, 473 


. 

Subject Index 


examples, 451,457,473,493,497, 

503, 510, 517, 570, 748 

Sweat, 215 

examples, 214,261, 475 

University, 729 

examples, 713, 729, 731 

Akaike Information Criterion (A1C), 

interpretation of, 545 

Welder, 245 

386,397, 704 

population, 541-42 

Analysis of variance, multivariate: 

sample, 550-51 

example, 244 

one-way, 301 

tests of hypothesis in, 563-64 

Wheat, 571 

two-way, 315, 340 

variance explained, 561-62 

example, 570 

Analysis of variance, univariate: 

CART, 644 


one-way, 297 

Central-limit theorem, 176 


two-way, 312 

Characteristic equation, 97 


AN OVA (see Analysis of variance, 

Characteristic roots (see Eigenvalues) 


univariate) 

Characteristic vectors (see 


Autocorrelation, 414 

Eigenvectors) 


Autoregressive model, 415 

Chemoff faces, 27 


Average linkage (see Cluster analysis) 

Chi-square plots, 184 

Classification: 


Bayesian Information Criterion 

Anderson statistic, 592 


(BIC), 705 

Bayes’ rule, 584, 608 


Biplot, 726,730 

confusion matrix, 598 


Bonferroni intervals: 

error rates, 596,598, 599 


comparison with T 1 intervals, 234 

expected cost, 581, 607 


definition, 232 

Lachenbruch holdout procedure, 


for means, 232,276,291 

599, 619 


for treatment effects, 309, 317-18 

linear discriminant functions, 585, 


Box’s M test (see Covariance matrix, 

586,590,591,611,623 


test for equality of) 

with logistic regression, 638-39 
misclassification probabilities, 579- 


Canonical correlation analysis: 

80,583 

i 

canonical correlations, 539, 541, 

with normal populations, 584, 


547,551 

593,609 

i 

canonical variables, 539, 

quadratic discriminant function, 


541-42, 551 

594, 610 


correlation coefficients in, 546, 

qualitative variables, 644 

f 

551-52 

selection of variables, 648 

i 

definition of, 541,550 

for several groups, 606, 629 


errors of approximation, 558 

for two groups, 576, 584, 591 


geometry of, 549 

Classification trees, 644 
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Cluster analysis: 

algorithm, 681,696 
average linkage, 681,690 
complete linkage, 681, 685 
dendrogram, 681 
hierarchical, 680 
inversions in, 695 
AT-means, 696 
similarity and distance, 677 
similarity coefficients, 675,678 
single linkage, 681, 682 
with statistical models, 703 
Ward’s method, 692 
Coefficient of determination, 367, 403 
Communality, 484 
Complete linkage (see Cluster 
analysis) 

Confidence intervals: 

mean of normal population, 211 
simultaneous, 225, 232, 235,265, 

276,309,317-18 
Confidence regions: 
for contrasts, 281 
definition, 220 

for difference of mean vectors, 

286,292 

for mean vectors, 221 
for paired comparisons, 276 
Contingency table, 716 
Contrast matrix, 280 
Contrast vector, 279 
Control chart: 
definition, 239 
ellipse format, 241, 250, 460 
for subsample means, 249, 251 
multivariate, 241, 461-62,465 
T 2 chart, 243,248,250,251,462 
Control regions: 
definition, 247 

for future observations, 247, 251, 

463 

Correlation: 

autocorrelation, 414 
coefficient of, 8, 71 
geometrical interpretation of 
sample, 119 
multiple, 367,403,548 


partial, 409 
sample, 8,117 
Correlation matrix: 
population, 72 
sample, 9 

tests of hypotheses for 
equicorrelation, 457-58 
Correspondence analysis: 
algebraic development, 718 
correspondence matrix, 718 
inertia, 716,717,725 
matrix approximation method, 724 
profile approximation method, 724 
Correspondence matrix, 718 
Covariance: 
definitions of, 69 
of linear combinations, 75, 76 
sample, 8 
Covariance matrix: 
definitions of, 69 
distribution of, 175 
factor analysis models for, 483 
geometrical interpretation of 
sample, 119,124-26 
large sample behavior, 175 
as matrix operation, 139 
partitioning, 73, 78 
population, 71 
sample, 123 
test for equality of, 310 

Data mining: 

lift chart, 742 
model assessment, 742 
process, 741 
Dendrogram, 681 
Descriptive statistics: 
correlation coefficient, 8 
covariance, 8 
mean, 7 
variance, 7 

Design matrix, 362,388,411 
Determinant: 
computation of, 93 
product of eigenvalues, 104 

Discriminant function (see 
Classification) 


] 


Distance: 

Canberra, 674 
Czekanowski, 674 
development of, 30-37, 64 
Euclidean, 30 
Minkowski, 673 
properties, 37 
statistical, 31,36 
Distributions: 
chi-square (table), 760 
F (table), 761,762,763 
multinomial, 264 
normal (table), 758 
Q-Q plot correlation coefficient 
(table), 181 
t (table), 759 
Wishart, 174 

Eigenvalues, 97 
Eigenvectors, 98 
EM algorithm, 252 
Estimation: 

generalized least squares, 422 
least squares, 364 
maximum likelihood, 168 
minimum variance, 369-70 
unbiased, 121,123,369-70 
weighted least squares, 420 
Estimator (see Estimation) 

Expected value, 67, 68 
Experimental unit, 5 

Factor analysis: 
bipolar factor, 506 
common factors, 482, 483 
communality, 484 
computational details, 527 
of correlation matrix, 490,494,529 
Heywood cases, 497, 529 
least squares (Bartlett) computation 
of factor scores, 514,515 
loadings, 482,483 
maximum likelihood estimation 
in, 495 

nonuniqueness of loadings, 487 
oblique rotation, 506,512 
orthogonal factor model, 483 





principal component estimation 
in, 488, 490 

principal factor estimation in, 494 
regression computation of factor 
scores, 516, 517 
residual matrix, 490 
rotation of factors, 504 
specific factors, 482,483 
specific variance, 484 
strategy for, 520 
testing for the number of 
factors, 501 
varimax criterion, 507 
Factor loading matrix, 482 
Factor scores, 515, 517 
Fisher’s linear discriminants: 
population, 654 
sample, 590-91,623 
scaling, 589 

Gamma plot, 184 
Gauss (Markov) theorem, 369 
Generalized inverse, 369, 421 
Generalized least squares (see 
Estimation) 

Generalized variance: 
geometric interpretation of sample, 
124,135-36 
sample, 123,135 
situations where zero, 133 
General linear model: 
design matrix for, 362,388 
multivariate, 388 
univariate, 362 
Geometry: 
of classification, 618 
generalized variance, 124,135-36 
of least squares, 367 
of principal components, 468, 469 
of sample, 119 
Gram-Schmidt process, 86 
Graphical techniques: 
biplot, 726,730 
Chemoff faces, 27 
marginal dot diagrams, 12 
n points in p dimensions, 17 
p points in n dimensions, 19 
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Graphical techniques ( continued) 

scatter diagram (plot), 11, 20 
stars, 26 

Growth curve, 24, 328 

Hat matrix, 364, 421, 643 

Hey wood cases (see Factor analysis) 

Hotelling’s T 2 (see 7*-statistic) 

Independence: 

definition, 69 

of multivariate normal variables, 
159-60 

of sample mean and covariance 
matrix, 174 

tests of hypotheses for, 472 
Inequalities: 

Cauchy-Schwarz, 78 
extended Cauchy-Schwarz, 79 

Inertia, 725 

Influential observations, 384, 643 
Invariance of maximum likelihood 
estimators, 172 
Item (individual), 5 

K-means (see Cluster analysis) 

Lawley-Hotelling trace statistic, 

336, 398 

Leverage, 381,384 
Lift chart, 742 
Likelihood function, 168 
Likelihood ratio tests: 

definition, 219 
limiting distribution, 220 
in regression, 374,396 
and 7*, 218 

Linear combination of vectors, 

83,165 

Linear combination of variables: 

mean of, 76 

normal populations, 156,157 
sample covariances of, 141,144 
sample means of, 141,144 
variance and covariances of, 76 
Logistic classification: 
classification rule, 638-39 


linear discriminant, 639 
Logistic regression: 
deviance, 642 
estimation in, 637-38 
logit, 635 
logistic curve, 636 
model, 637 
residuals, 643 

tests of regression coefficients, 638 

MANOVA (see Analysis of variance, 
multivariate) 

Matrices: 

addition of, 88 
characteristic equation of, 97 
correspondence, 718 
definition of, 54, 87 
determinant of, 93,104 
dimension of, 88 

eigenvalues of, 59, 97, 98 \ 

eigenvectors of, 59, 98 
generalized inverses of, 364, 

369, 421 
identity, 58,90 
inverses of, 58, 95 
multiplication of, 56, 90,109 
orthogonal, 59, 97 
partitioned, 73, 74, 78 
positive definite, 61,62 
products of, 56, 90, 91 

random, 66 j 

rank of, 94 

scalar multiplication in, 89 
singular and nonsingular, 95 
singular-value decomposition, 100, 
721,725,728 

spectral decomposition, 61, 100 \ 

square root, 66 ! 

symmetric, 57,90 : 

trace of, 96 I 

transpose of, 55,89 
Maxima and minima (with matrices), 

79,80 \ 

Maximum likelihood estimation: j 

development, 170-72 
invariance property of, 172 
in regression, 370, 395, 404-05 


Mean, 66 
Mean vector 

definition, 69 
distribution of, 174 
large sample behavior, 175 
as matrix operation, 139 
partitioning, 73,78 
sample, 9, 78 

Minimal spanning tree, 715 
Missing observations, 251 
Mixture model, 703 
Model based clustering: 

estimation in, 704 
mixture model, 703 
model selection, 704-05 
Model selection criterion: 

AIC, 386, 397, 704 
BIC, 705 

Multicollinearity, 386 
Multidimensional scaling: 

algorithm, 709 
development, 706-15 
sstress, 709 
stress, 708 

Multiple comparisons (see 
Simultaneous confidence 
intervals) 

Multiple correlation coefficient: 

population, 403, 548 
sample, 367 

Multiple regression (see Regression 
and General linear model) 
Multivariate analysis of variance (see 
Analysis of variance, multivariate) 
Multivariate control chart (see 
Control chart) 

Multivariate normal distribution (see 
Normal distribution, multivariate) 

Neural network, 647 
Nonlinear mapping, 715 
Nonlinear ordination, 738 
Normal distribution: 

bivariate, 151 

checking for normality, 177 

conditional, 160-61 

constant density contours, 153, 435 


marginal, 156,158 
maximum likelihood estimation 
in, 171 

multivariate, 149-55 
properties of, 156-67 
transformations to, 192 

Normal equations, 421 
Normal probability plots (see Q-Q 
plots) 

Outliers: 

definition, 187 
detection of, 189 

Paired comparisons, 273-79 
Partial correlation, 409 
Partitioned matrix: 

definition, 73, 74, 78 
determinant of, 202-03 
inverse of, 203 

Pillai’s trace statistic, 336, 398 
Plots: 

biplot, 726 

biplot, alternative, 730-31 
C p , 385 

factor scores, 515, 517 
gamma (or chi-square), 184 
principal components, 454-55 
Q-Q, 178,382 
residual, 382-83 
scree, 445 

Positive definite (see Quadratic forms) 
Posterior probabilities, 584, 608 
Principal component analysis: 

correlation coefficients in, 433, 
442,451 

for correlation matrix, 437, 451 
definition of, 431-32,442 
equicorrelation matrix, 440-41 
geometry of, 466-70 
interpretation of, 435-36 
large-sample theory of, 456-69 
monitoring quality with, 459-65 
plots, 454-55 
population, 431-41 
reduction of dimensionality by, 
466-68 
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Principal component analysis 
(continued) 
sample, 441-53 

tests of hypotheses in, 457-59,472 
variance explained, 433, 437,451 

Procustus analysis: 

development, 732-39 
measure of agreement, 733 
rotation, 733 

Profile analysis, 323-28 
Proportions; 

large-sample inferences, 264-65 
multinomial distribution, 264 

Q-Q plots; 

correlation coefficient, 181 
critical values, 181 
description, 177-82 
Quadratic forms: 
definition, 62, 99 
extrema, 80 

nonnegative definite, 62 
positive definite, 61, 62 

Random matrix, 66 
Random sample, 119-20 
Regression (see also General linear 
model): 

autoregressive model, 415 
assumptions, 361-62, 370, 388, 395 
coefficient of determination, 

367,403 

confidence regions in, 371, 378, 
399, 421 
C p plot, 385 

decomposition of sum of squares, 
366-67, 389 

extra sum of squares and cross 
products, 374,396 
fitted values, 364, 389 
forecast errors in, 379 
Gauss theorem in, 369 
geometric interpretation of, 367 
least squares estimates, 364, 393 
likelihood ratio tests in, 374, 396 
maximum likelihood estimation in, 
370-71,395,404, 407 


multivariate, 387-401 
regression coefficients, 364,406 
regression function, 370,404 
residual analysis in, 381-83 
residuals, 364, 381, 389 
residual sum of squares and cross 
products, 364,389 
sampling properties of estimators, 
369-71,393,395 
selection of variables, 385-86 
univariate, 360-62 
weighted least squares, 420 
with time-dependent errors, 413-17 
Regression coefficients (see 
Regression) 

Repeated measures designs, 279-83, 
328-32 

Residuals, 364, 381-83, 389,455, 643 

Roy’s largest root, 336, 398 

Sample: 

geometry, 119 

Sample splitting, 520, 599,742 
Scree plot, 445 

Simultaneous confidence ellipses: 

as projections, 258-60 
Simultaneous confidence intervals: 

comparisons of, 229-31,234,238 
for components of mean vectors, 
225,232,235 
for contrasts, 281 
development, 223-26 
for differences in mean vectors, 
288,291-92 

for paired comparisons, 276 
as projections, 258 
for regression coefficients, 371 
for treatment effects, 309, 

317-18 

Single linkage (see Cluster analysis) 
Singular matrix, 95 
Singular-value decomposition, 100, 
721,725,728 

Special causes (of variation), 239 
Specific variance, 484 
Spectral decomposition, 61,100 
SStress, 709 


Standard deviation: 

population, 72 
sample, 7 

Standard deviation matrix: 

population, 72 
sample, 139 

Standardized observations, 8,449 
Standardized variables, 436 
Stars, 26 

Strategy for multivariate 
comparisons, 337 
Stress, 708 

Studentized residuals, 381 
Sufficient statistics, 173 
Sum of squares and cross products 
matrices: 

between, 302 
total, 302 
within, 302 

lime dependence (in multivariate 
observations), 256-57, 413-17 
r 2 -statistia 

definition of, 211-13 
distribution of, 212 
invariance property of, 215-16 
in quality control, 243,247-48, 250- 
51,462 

in profile analysis, 324, 325 
for repeated measures designs, 280 
single-sample, 211-12 
two-sample, 286 
two-sample, approximate, 294 


Trace of a matrix, 96 
Transformations of data, 192-200 

Variables: 

canonical, 541-42, 550-51 
dummy, 363 
predictor, 361 
response, 361 
standardized, 436 
Variance: 
definition, 68 
generalized, 123,134 
geometrical interpretation of, 119 
total sample, 137,442,451, 561 
Varimax rotation criterion, 507 
Vectors: 
addition, 51, 83 
angle between, 52, 85 
basis, 84 

definition of; 49, 82 
inner product, 52, 53, 85 
length of, 51, 53, 84 
linearly dependent, 53,83 
linearly independent, 53, 83 
linear span, 83 

perpendicular (orthogonal), 53,86 
projection of, 54, 86, 87 
random, 66 

scalar multiplication, 50, 82 
unit, 51 

vector space, 83 

Wilks’s lambda, 217,303, 398 

Wishart distribution, 174 








